search

whale2 / async-kinesis-client

Python Kinesis Client library utilising asyncio
MIT License
13 stars 4 forks source link

Sequence Number Mismatch #3

Closed scattym closed 5 years ago

scattym commented 5 years ago

Hi,

I've been doing some failure testing around checkpointing and somehow I have got into a state where by dynamodb has a sequence number recorded that kinesis doesn't know about.

I was getting this with 0.2.10 and just upgraded to the latest git release but I am still getting the same error. Note: I didn't clear the error after upgrading. I just upgraded to see if this is an error condition your latest changes address.

Any thoughts as to what is going on?


  File "/home/matt/pycharm-2019.1.2/helpers/pydev/pydevd.py", line 1758, in <module>
    main()
  File "/home/matt/pycharm-2019.1.2/helpers/pydev/pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/matt/pycharm-2019.1.2/helpers/pydev/pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/matt/pycharm-2019.1.2/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/matt/git/python-tool/scripts/akcl", line 42, in <module>
    asyncio.get_event_loop().run_until_complete(read_stream())
  File "/home/matt/miniconda3/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/home/matt/git/python-tool/scripts/akcl", line 24, in read_stream
    async for shard_reader in consumer.get_shard_readers():
  File "/home/matt/git/async-kinesis-client/src/async_kinesis_client/kinesis_consumer.py", line 381, in get_shard_readers
    **iterator_args
  File "/home/matt/VirtualEnv/CPython3.7/lib/python3.7/site-packages/aioboto3/aiobotocore/client.py", line 105, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidArgumentException: An error occurred (InvalidArgumentException) when calling the GetShardIterator operation: StartingSequenceNumber 4959578909659155314359522313291951848983041376358760466 used in GetShardIterator on shard shardId-000000000001 in stream data-stream under account 11111111 is invalid.```
whale2 commented 5 years ago

Just encountered it myself yesterday, no idea so far. First, I'm going to implement fall-back to AT_TIMESTAMP, so it won't be a critical error and after that find out why it happens in the first place. Thanks for reporting.

whale2 commented 5 years ago

Fixed in 0.2.11 The reason was following - sequence number is being split in two parts to avoid string comparison in DynamoDB. In cases when second part should have contained leading zero, that leading zero was missing. Now I store complete sequence number in DynamoDB for recovering.

scattym commented 5 years ago

Awesome, nice work. Thanks for the quick response :)