Not receiving ion_text and can't parse ion_binary in QLDB session

[rafayet-monon]

Not receiving ion_text and can't parse ion_binary in QLDB session while executing query

I am fairly new in QLDB. This might not be a issue as I don't have much knowledge in QLDB. I have a ledger named hk-ledger-test and a table named testing. I am trying to execute a query in that table.

gem name: aws-sdk-qldbsession

Version of Ruby: 2.4.1, OS environment : MacOs

Code snippets

I have the following method to execute query

def test
      cred = Aws::Credentials.new(ENV['AWS_ACCESS_KEY'], ENV['AWS_SECRET_ACCESS_KEY'])

      qldb_session_client = Aws::QLDBSession::Client.new(region: 'ap-southeast-1', credentials: cred)

      start_session_req = Aws::QLDBSession::Types::StartSessionRequest.new
      start_session_req.ledger_name = 'hk-ledger-test'

      command_request = Aws::QLDBSession::Types::SendCommandRequest.new

      command_request.start_session = start_session_req

      resp = qldb_session_client.send_command(command_request)
      session_token = resp.start_session.session_token

      command_request = Aws::QLDBSession::Types::SendCommandRequest.new
      command_request.session_token = session_token
      command_request.start_transaction = Aws::QLDBSession::Types::StartTransactionRequest.new
      resp = qldb_session_client.send_command(command_request)
      transaction_id = resp.start_transaction.transaction_id

      command_request = Aws::QLDBSession::Types::SendCommandRequest.new
      command_request.session_token = session_token
      command_request.execute_statement = Aws::QLDBSession::Types::ExecuteStatementRequest.new
      command_request.execute_statement.transaction_id = transaction_id
      command_request.execute_statement.statement = 'select * from testing'
      resp = qldb_session_client.send_command(command_request)
      resp
    end

And it's returning the following Struct, ion_text was supposed to be in there but it's showing nil

#<struct Aws::QLDBSession::Types::SendCommandResult start_session=nil, start_transaction=nil, end_session=nil, commit_transaction=nil, abort_transaction=nil, execute_statement=#<struct Aws::QLDBSession::Types::ExecuteStatementResult first_page=#<struct Aws::QLDBSession::Types::Page values=[#<struct Aws::QLDBSession::Types::ValueHolder ion_binary="\xE0\x01\x00\xEA\xEE\x9A\x81\x83\xDE\x96\x87\xBE\x93\x89firstname\x88lastname\xDE\x9D\x8A\x8Dtesting first\x8B\x8Ctesting last", ion_text=nil>, #<struct Aws::QLDBSession::Types::ValueHolder
ion_binary="\xE0\x01\x00\xEA\xEE\x9A\x81\x83\xDE\x96\x87\xBE\x93\x89firstname\x88lastname\xDE\x9D\x8A\x8Dtesting first\x8B\x8Ctesting last", ion_text=nil>, #<structAws::QLDBSession::Types::ValueHolder ion_binary="\xE0\x01\x00\xEA\xEE\x9A\x81\x83\xDE\x96\x87\xBE\x93\x89firstname\x88lastname\xDB\x8A\x83ami\x8B\x84boka", ion_text=nil>], next_page_token=nil>>, fetch_page=nil>

I don't know if I am doing something wrong or if this is the right way. I was then trying to decode the binary -

string_io = StringIO.new(resp.execute_statement.first_page.values[0].ion_binary).binmode
decoded_string= Aws::EventStream::Decoder.new.decode(string_io)

it shows #<Enumerator: #<Enumerator::Generator:0x007fc9598a2350>:each>

where do I go from here? How do I get the data?

[mullermp]

Hey there - have you checked the docs for available methods you can use? https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/QLDB/Client.html

I see you're trying to use raw types .. such as StartTransactionRequest etc.. the library does all of that on your behalf.

[rafayet-monon]

I see you're trying to use raw types .. such as StartTransactionRequest etc.. the library does all of that on your behalf.

Hi, yeah I understand your concern, you have a valid point. So I change it like below and I am trying to work with QLDBSession -

class Qldbsession
    attr_accessor :client, :ledger_name

    def initialize
      cred = Aws::Credentials.new(ENV['AWS_ACCESS_KEY'], ENV['AWS_SECRET_ACCESS_KEY'])
      @client = Aws::QLDBSession::Client.new(region: 'ap-southeast-1', credentials: cred)
      @ledger_name = 'hk-ledger-test'
    end

    def send_execute_command
      session_token = start_session
      transaction_id = start_transaction session_token
      resp = client.send_command({
                                     session_token: session_token,
                                     execute_statement: {
                                         transaction_id: transaction_id, # required
                                         statement: 'select * from testing', # required
                                     }
                                 })

      resp
    end

    private

    def start_session
      resp = client.send_command({
                                     start_session: {
                                         ledger_name: ledger_name
                                     }
                                 })

      resp.start_session.session_token
    end

    def start_transaction session_token
      resp = client.send_command(session_token: session_token,
                                 start_transaction: {
                                 })

      resp.start_transaction.transaction_id
    end

  end

But the result still is the same - ion_text is nil

#<struct Aws::QLDBSession::Types::SendCommandResult start_session=nil, start_transaction=nil, end_session=nil, commit_transaction=nil, abort_transaction=nil, execute_statement=#<struct Aws::QLDBSession::Types::ExecuteStatementResult first_page=#<struct Aws::QLDBSession::Types::Page values=[#<struct Aws::QLDBSession::Types::ValueHolder ion_binary="\xE0\x01\x00\xEA\xEE\x9A\x81\x83\xDE\x96\x87\xBE\x93\x89firstname\x88lastname\xDE\x9D\x8A\x8Dtesting first\x8B\x8Ctesting last", ion_text=nil>, #<struct Aws::QLDBSession::Types::ValueHolder ion_binary="\xE0\x01\x00\xEA\xEE\x9A\x81\x83\xDE\x96\x87\xBE\x93\x89firstname\x88lastname\xDE\x9D\x8A\x8Dtesting first\x8B\x8Ctesting last", ion_text=nil>, #<struct Aws::QLDBSession::Types::ValueHolder ion_binary="\xE0\x01\x00\xEA\xEE\x9A\x81\x83\xDE\x96\x87\xBE\x93\x89firstname\x88lastname\xDB\x8A\x83ami\x8B\x84boka", ion_text=nil>], next_page_token=nil>>, fetch_page=nil>

[mullermp]

Sorry -- I see you're using QLDBSession and not QLDB! To be perfectly honest, I've never used either service personally so I'll try my best to help you here. It looks like what you're trying to do is correct as per the docs. I tried doing this myself with populated sample data from the console. When I perform my query (on the sample Person table) I get back a list of ValueHolder data:

#<struct Aws::QLDBSession::Types::ValueHolder
        ion_binary=
         "\xE0\x01\x00\xEA\xEE\xB6\x81\x83\xDE\xB2\x87\xBE\xAF\x89FirstName\x88LastName\x83DOB\x85GovId\x89GovIdType\x87Address\xDE\xE1\x8A\x85Brent\x8B\x85Logan\x8Ce\xC0\x0F\xAF\x87\x83\x8D\x8BLOGANB486CG\x8E\x8E\x8EDriver License\x8F\x8E\xAB43 Stockert Hollow Road, Everett, WA, 98203",
        ion_text=nil>,

It looks like ion_text is supposed to be nil..? But ion_binary does include everything, and in your example I see ... testing first\x8B\x8Ctesting last so perhaps this is just a design decision with their service?

To access that data iteratively, I did something like the following:

resp.execute_statement.first_page.values.each do |v|
  puts v.ion_binary
end  

I'm thinking there is a way to parse this in a sane way, let me ask around internally and find out.

[mullermp]

Looking here: http://amzn.github.io/ion-docs/docs/binary.html there appears to not be an ion parser for Ruby.

[mullermp]

The reason ion_text exists is that Request and Response objects share a similar structure, and the Request allows for ion text to be passed in to the service for simplicity, but the response doesn't return it.

[mullermp]

I've engaged relevant teams to start a feature request for Ion support in Ruby. There may be a quick and dirty way to do this using the C extension with a Ruby wrapper.

[rafayet-monon]

Thanks a lot for the quick and supportive response. I will be eagerly waiting for the feature.

[mullermp]

As a work around for now, if you need to use the QLDB service, you could use the SDK in one of these languages documented here: http://amzn.github.io/ion-docs/. Alternatively you can use the Ruby SDK and perhaps use system calls to call the ion library in python or C.

[rafayet-monon]

In the meantime I will try to do one of that. Thanks.

[mullermp]

Going to close this issue for now. The Ion Ruby package was setup and is in a private GitHub repo to be developed on.

[rafayet-monon]

Still in development I guess?

[mullermp]

Yeah, it's still in development unfortunately. I wouldn't hold my breath on it to be honest :/ it sounds like it's not highest priority for them. I can check with the Ion team on what plans they have for it.

[magnum]

@rafayet-monon: ion decoding apart, did you manage to complete the transaction? your code above returns results, but for example if someone needs insert, i suppose that a final commit_transaction request is needed https://docs.aws.amazon.com/qldb/latest/developerguide/API_QLDB-Session_SendCommand.html#qldb-QLDB-Session_SendCommand-response-CommitTransaction with a CommitDigest https://docs.aws.amazon.com/qldb/latest/developerguide/API_QLDB-Session_CommitTransactionResult.html#qldb-Type-QLDB-Session_CommitTransactionResult-CommitDigest but how to calculate it?

[akhmel]

@mullermp Please let us know if there are any workarounds for this ? how ion_binary string can be decoded in ruby ? For example: "\xE0\x01\x00\xEA\xEE\x99\x81\x83\xDE\x95\x87\xBE\x92\x85email\x8Bcustomer_id\xDE\xC2\x8A\x8E\xA3andrey.khmelevsky+1@wearepeachy.com\x8B&\a \x9C\x1E\x81A\x84\x8E\x91Andrey Revision 2"

[mullermp]

As far as I know, you'd have to take the data and use it with one of the available languages on https://amazon-ion.github.io/ion-docs/index.html. There is a C library and in theory a ruby gem can be made from an extension. I can check in with the Ion team and see if Ruby is still on their roadmap.

[akhmel]

@mullermp thank you, much appreciated.

[mullermp]

The Ion team has this planned (TM) because they would like to support all of the AWS SDK languages. I cannot give you any timelines. If you need something immediately, I recommend using one of the other language tools and pumping output to it somehow, maybe using system calls or backticks..