Protobuf metadata

[NRHelmi]

This PR adds support to:

• Metadata protobuf

  relations {
  relation_id {
  arguments {
    tag: CONSTANT_TYPE
    constant_type {
      rel_type {
        tag: PRIMITIVE_TYPE
        primitive_type: SYMBOL
      }
      value {
        arguments {
          tag: SYMBOL
          string_val: "output"
        }
      }
    }
  }
  arguments {
    tag: PRIMITIVE_TYPE
    primitive_type: INT_64
  }
  }
  file_name: "0.arrow"
  }

• Structured v2 response

  {'transaction': {'id': '3d1588b0-042a-f244-76ce-c4a5e38b76d3', 'response_format_version': '2.0.3', 'state': 'COMPLETED'}, 'metadata': relations {
  relation_id {
  arguments {
    tag: CONSTANT_TYPE
    constant_type {
      rel_type {
        tag: PRIMITIVE_TYPE
        primitive_type: SYMBOL
      }
      value {
        arguments {
          tag: SYMBOL
          string_val: "output"
        }
      }
    }
  }
  arguments {
    tag: PRIMITIVE_TYPE
    primitive_type: INT_64
  }
  }
  file_name: "0.arrow"
  }
  , 'results': [{'relationId': '/:output/Int64', 'table': pyarrow.Table
  v1: int64 not null
  ----
  v1: [[1]]}], 'problems': []}

• v2 show results

  
  /:output/Int64/Int64
  (1, 200)
  (2, 300)

/:output/String ('hi',)

/:output/Int64 (1,) (2,) (3,)

* transaction async integration tests added
* CI github workflow
* flake8 linter

[denisgursky]

Hm, string_val: "output" is comes out as a string?

[denisgursky]

Does it make sense to include metadata as a part of results? Basically the same thing as in https://github.com/RelationalAI/rai-sdk-javascript/pull/44

[NRHelmi]

Hm, string_val: "output" is comes out as a string?

I believe it's printed correctly because no json serialization performed here

[NHDaly]

Hm, string_val: "output" is comes out as a string?

are we using python2 or python3? The spec says that bytes should become a "bytes" type in Python3: bytes (Python 3) https://developers.google.com/protocol-buffers/docs/proto3#scalar

[NRHelmi]

Hm, string_val: "output" is comes out as a string?

are we using python2 or python3? The spec says that bytes should become a "bytes" type in Python3: bytes (Python 3) https://developers.google.com/protocol-buffers/docs/proto3#scalar

The SDK uses python 3, python 2 was deprecated on January 1, 2020 :cry:

[NRHelmi]

Does it make sense to include metadata as a part of results? Basically the same thing as in RelationalAI/rai-sdk-javascript#44

@denisgursky since I did also some code refactoring in this PR I think it would be better to address this change in a separate PR (probably when working on ResultTable for this SDK) for the sake of simplicity :pray:

[NRHelmi]

if no other updates are required, I'll merge by the end of the day thank you :pray:

[larf311]

In the future, please separate out code formatting changes into their own PR. It makes it next to impossible to review the actual changes.

[NRHelmi]

In the future, please separate out code formatting changes into their own PR. It makes it next to impossible to review the actual changes.

Yeah sorry for that :pray: , didn't notice that this PR is getting really complicated to review, maybe a quick guideline on how to review this PR could help:

• For protobuf metadata:
  • message_pb2.py and schema_pb2.py protobuf generated code are under railib/pb.
  • please check _parse_metadata_proto function under api.py: this function deserializes the byte array received into MetadataInfo object.
  • get_transaction_metadata function under api.py returns the metadata proto content (we removed the old get_transaction_metadata, similar behavior but the new one sends in headers {"Accept": "application/x-protobuf"}.
  • TransactionAsyncResponse class under api.py represents the v2 response object {transaction: ..., metadata:..., results: ..., problems: ...}
• Multipart parts are converted into TransactionAsyncFile object {name: ..., filename: ..., content_type: ..., content: ...} contents. (please notice that
• Integration tests under tests/integration.py contains a test for transaction, metadata proto, problems and arrow results
• CI github workflow is under .github/workflows/build.yaml: should install deps, lint using flake8 and run integration tests

Please ignore the other changes since they are mainly related to code formatting Thank you :pray:

[larf311]

Installing from source and trying to run the examples gives me:

➜  rai-sdk-python git:(hnr-metadata-proto) python3 ./examples/get_transaction_metadata.py cf686a5e-7b8f-54c8-ff2b-d8ebdb68618d
Traceback (most recent call last):
  File "/Users/trevorpaddock/Dev/rai-sdk-python/./examples/get_transaction_metadata.py", line 19, in <module>
    from railib import api, config, show
  File "<frozen zipimport>", line 259, in load_module
  File "/usr/local/lib/python3.9/site-packages/rai_sdk-0.6.4-py3.9.egg/railib/api.py", line 24, in <module>
ModuleNotFoundError: No module named 'requests_toolbelt

I need to run:

python3 -m pip install -r requirements.txt

Then everything works. Do we need to add that to the README or do you need to update setup.py?

[NRHelmi]

Thanks @larf311 just updated the README file, it was missing the dependencies installation

pip install -r requirements.txt

[NRHelmi]

If no other updates are required, I'll merge this PR by the end of the day thank you :pray: