`v3/filter` endpoint: Paged responses return no symbology data for a subset of the queried response pages

[davidandreoletti]

Context: Client application with authenticated with API key.

2 issues with requests having couple thousands results:

• some HTTP responses but not all (when iterating over the next contains 0 symbols. See BLOCK A in the sample response.
• HTTP responses with total result under 15k are not all returned. BLOCK B in the sample response is the last page retrievable. At this point: -- ~500 symbols where retrieved out of expected 7942. -- ~70 pages consumed out of max 150 page limit

Request

https://api.openfigi.com/v3/filter
parameters: {
  "micCode": "XNMS",
   "marketSecDes": "Equity"
}

Sample response

.... **BLOCK A**
<Response [200 OK]>
{'data': [{'figi': 'BBG00W9MJKF0', 'name': 'GEMINI THERAPEUTICS INC', 'ticker': 'GMTX', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00W9MJK79', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00W9MJL04', 'securityType2': 'Common Stock', 'securityDescription': 'GMTX'}, {'figi': 'BBG00WBPG4K3', 'name': 'HARMONY BIOSCIENCES HOLDINGS', 'ticker': 'HRMY', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00WBPG4C2', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00WBPG548', 'securityType2': 'Common Stock', 'securityDescription': 'HRMY'}, {'figi': 'BBG00WFBY5R0', 'name': 'NANO-X IMAGING LTD', 'ticker': 'NNOX', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00RDSZ6J2', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00RDSZ6K0', 'securityType2': 'Common Stock', 'securityDescription': 'NNOX'}, {'figi': 'BBG00WFDP0Q8', 'name': 'KYMERA THERAPEUTICS INC', 'ticker': 'KYMR', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00SDHC8D1', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00SDHC8F9', 'securityType2': 'Common Stock', 'securityDescription': 'KYMR'}, {'figi': 'BBG00WRF65R9', 'name': 'UNIVERSE PHARMACEUTICALS INC', 'ticker': 'UPC', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00WRF65K6', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00WRF66B4', 'securityType2': 'Common Stock', 'securityDescription': 'UPC'}, {'figi': 'BBG00WWF0QQ4', 'name': 'FUSION FUEL GREEN PLC-A', 'ticker': 'HTOO', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00VZRQFB9', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00VZRQFC8', 'securityType2': 'Common Stock', 'securityDescription': 'HTOO'}, {'figi': 'BBG00WXXSW36', 'name': 'METACRINE INC', 'ticker': 'MTCR', 'exchCode': 'UQ', 'compositeFIGI': 'BBG009R3Z5S4', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG009R3Z5T3', 'securityType2': 'Common Stock', 'securityDescription': 'MTCR'}], 'next': 'QW9Fc1FrSkhNREJYV0ZoVFZ6Y3kgNjM=.KvA7GTys1c3Zxh2r/FreUfqgnuD7W/cYqJzZ4hFSFGY=', 'total': 7942}
<Response [200 OK]>
{'data': [], 'next': 'QW9Fc1FrSkhNREJZTVUweE1EZzMgNjQ=.fnXeVH+PcwPBN4SB3dR4NprYeTKGfJ8ej1zN3KXyp1g=', 'total': 7942}
<Response [200 OK]>
{'data': [], 'next': 'QW9Fc1FrSkhNREJZTVZBeE1FZ3cgNjU=.TADybkuQWKQG1yIR7E3lPIpLoOIQYHJBW7rSCuMYf7Y=', 'total': 7942}
<Response [200 OK]>
{'data': [], 'next': 'QW9Fc1FrSkhNREJZTVZBNFExTTEgNjY=.CLHeR7D9w1bmO9WLk+X4mwni2uojjBmcrWMnvvSEe9Q=', 'total': 7942}
<Response [200 OK]>
{'data': [], 'next': 'QW9Fc1FrSkhNREJZTVZCS1dsTXkgNjc=.pihvjZW8EhBBabqlXASKoFFjXLqidZdPY+W//er0rOQ=', 'total': 7942}
<Response [200 OK]>
{'data': [], 'next': 'QW9Fc1FrSkhNREJZTVZFNFNEQXkgNjg=.gTtTZBMl463PaCXlHf9SfhZKWnuiWxAQtDlqJeOq0Rs=', 'total': 7942}
<Response [200 OK]>
{'data': [], 'next': 'QW9Fc1FrSkhNREJZTVZOR05GRXggNjk=.Qd320bMU4Y+1QCJKYmVlFsnr3m1lpXI0aqEfVP3DXLY=', 'total': 7942}
<Response [200 OK]>
{'data': [], 'next': 'QW9Fc1FrSkhNREJZTVZOUVZESXggNzA=.0l/a/5QmFr8uMzl81LDQfemmgrOliOjf9A0ONqUdrRI=', 'total': 7942}
<Response [200 OK]>
{'data': [], 'next': 'QW9Fc1FrSkhNREJZTVZOYVdWUXcgNzE=.teWblLvOQLezxXgW3IMxKGjfEPiWkhwsT9d5yJwPv8Q=', 'total': 7942}
<Response [200 OK]>
{'data': [{'figi': 'BBG00X3GR1T5', 'name': 'SUMMIT THERAPEUTICS INC', 'ticker': 'SMMT', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00X3GQL76', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00X3GQL94', 'securityType2': 'Common Stock', 'securityDescription': 'SMMT'}, {'figi': 'BBG00X5FVRL1', 'name': 'GRAYBUG VISION INC', 'ticker': 'GRAY', 'exchCode': 'UQ', 'compositeFIGI': 'BBG0088HHJ97', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG0088HHJB4', 'securityType2': 'Common Stock', 'securityDescription': 'GRAY'}, {'figi': 'BBG00X5FXL69', 'name': 'PMV PHARMACEUTICALS INC', 'ticker': '1202456D', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00X5FXL05', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG007NM4862', 'securityType2': 'Common Stock', 'securityDescription': '1202456D'}, {'figi': 'BBG00X9DZT61', 'name': 'ONCORUS INC', 'ticker': 'ONCR', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00DDYJ7N1', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00DDYJ7P9', 'securityType2': 'Common Stock', 'securityDescription': 'ONCR'}, {'figi': 'BBG00XCD19L1', 'name': 'AZIYO BIOLOGICS INC-CLASS A', 'ticker': 'AZYO', 'exchCode': 'UQ', 'compositeFIGI': 'BBG00BRBB838', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG00BRBB847', 'securityType2': 'Common Stock', 'securityDescription': 'AZYO'}, {'figi': 'BBG00XK4JQH0', 'name': 'ARRAY TECHNOLOGIES INC', 'ticker': 'ARRY', 'exchCode': 'UQ', 'compositeFIGI': 'BBG001V4SF46', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG002MX7TQ9', 'securityType2': 'Common Stock', 'securityDescription': 'ARRY'}], 'next': 'QW9Fc1FrSkhNREJZU3pSS1VWTTQgNzI=.g74pWALMBSGKBB4xxSj8BlmLN+DU6Iye+DanzjJT9UQ=', 'total': 7942}
.... **BLOCK B**
{'data': [{'figi': 'BBG011CK9Q65', 'name': 'OCEAN BIOMEDICAL INC', 'ticker': 'OCEA', 'exchCode': 'UQ', 'compositeFIGI': 'BBG011CK9Q01', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG011CK9QV7', 'securityType2': 'Common Stock', 'securityDescription': 'OCEA'}, {'figi': 'BBG011DY3218', 'name': 'SERA PROGNOSTICS INC', 'ticker': 'SERA', 'exchCode': 'UQ', 'compositeFIGI': 'BBG001J19ZJ7', 'securityType': 'Common Stock', 'marketSector': 'Equity', 'shareClassFIGI': 'BBG001V0FR24', 'securityType2': 'Common Stock', 'securityDescription': 'SERA'}], 'total': 7942}

[byang183]

Checking

[byang183]

I can not reproduce, would need running code to understand what's happening

[byang183]

with same filter query

1. with test finished locally, I confirmed that 150 pages returned,
2. all pages contains symbols, (though it is possible that page may contain less, and it might down to 0), this is kind a known limitation .

[byang183]

I got a total of 11297 figis back from that 150 pages of response

[davidandreoletti]

Thanks for the inputs.