`next` pagination not working as expected on `collection/manifest` endpoint

[himynamesdave]

See attempt to paginate

curl -X 'GET' \
  'http://127.0.0.1:8000/api/taxii2/cti_database/collections/mitre_attack_enterprise/manifest/?limit=10&match%5Btype%5D=attack-pattern&next=35941760' \
  -H 'accept: application/json' \
  -H 'Authorization: Basic XXXX'

https://www.awesomescreenshot.com/video/28346963?key=6b13876906858c4891038e540a999645

Is not working as detailed in spec

https://docs.oasis-open.org/cti/taxii/v2.1/os/taxii-v2.1-os.html#_Toc31107517

[fqrious]

I have a few questions... Was there up to a 30 seconds time difference between original request and this?

[fqrious]

You may need to change your --query.registry-ttl to a higher value

[himynamesdave]

Was there up to a 30 seconds time difference between original request and this?

I think this is the cause. I will document this. How would I go about modifying this value? If poss would be good to have this as an env var

However, pagination is still broken.

See

https://www.awesomescreenshot.com/video/28388944?key=40fa0bc87b0e1e031548ef01cb991cd4

I can page to page 2 (I can see objects in responses are different) HOWEVER, the value for next never changes, meaning I can't page to page 3.

[fqrious]

next goes to page 3... and then page 4 and then 5

cursor id doesn't change...

[himynamesdave]

I get it, but this poses 2 problems

1. the ttl issue, but more importantly
2. TAXII clients are typically expecting a different next value each time. I'm not saying this is right, but it is a reality.

We could just make the next value represent the page number, e.g. 2, 3, 4, etc if that would be simpler. That approach would solve the problem, be more user friendly, and conform to the spec (https://docs.oasis-open.org/cti/taxii/v2.1/os/taxii-v2.1-os.html#_Toc31107519)

[fqrious]

1. fixing the ttl issue is possible by using limit and offset in the query, however that means the database will have to reprocess all the already processed over and over again... This will be a problem when you have tens of thousands of objects
2. It doesn't actually say anything about it having to be different, but if you want it to be different . I can add "+{batchId}" to the next value

[himynamesdave]

1. understood
2. can we change the next value with each page (using batch id is fine), I think this will avoid user questions going fwd

[fqrious]

fixedd

[himynamesdave]

@fqrious i cannot get past page 2

https://www.awesomescreenshot.com/video/28604554?key=8dbc7f7893fa94fc6ce1d031c8254b4a

See page 1 -> 2 generated new objects

but page 2 -> 3 prints the same objects as page 2

[fqrious]

I can't reproduce this, that's why I added undef+random_float (instead of nextBatchId) to the cursor id for you...

I'll need a arangodb server with exact same setup/config to reproduce... This issue is not happening on the original 65.?.?.? server