search

biocommons / eutils

simplified searching, fetching, and parsing records from NCBI using their E-utilities interface
Apache License 2.0
58 stars 25 forks source link

.efetch() fails silently when batch query exceeds 250 IDs #171

Closed safay closed 10 months ago

safay commented 3 years ago 

In [29]: tx_accessions_ids =  ",".join(tx_for_test)

In [30]: tx_accessions_ids
Out[30]: 'NM_001366325.1,NM_032852.4,NM_145689.1,NM_001024070.1,NM_015687.3,NM_001370155.1,NM_001323004.1,NM_001354934.1,NM_018685.2,NM_031216.4,NM_030625.2,NM_153675.3,NM_001370507.1,NM_001322998.1,NM_002792.4,NM_177925.4,NM_001199240.1,NM_001314045.2,NM_198125.3,NM_001283017.2,NM_001378359.1,NM_005444.2,NM_003893.4,NM_001205248.2,NM_001033045.4,NM_001278464.1,NM_001206971.2,NM_001511.4,NM_001258345.2,NM_001260.2,NM_001352319.1,NM_001351522.2,NM_000551.3,NM_001300757.1,NM_001330721.1,NM_001012301.2,NM_000558.3,NM_001387137.1,NM_001109903.1,NM_173514.3,NM_001085469.2,NM_001289154.2,NM_001293075.2,NM_001261834.2,NM_001167903.1,NM_001353213.1,NM_024641.3,NM_153690.5,NM_007287.2,NM_004905.3,NM_001351049.1,NM_153236.3,NM_022336.3,NM_033127.3,NM_001164356.1,NM_001202470.3,NM_001308021.3,NM_001330407.1,NM_001288760.3,NM_022486.4,NM_001198754.2,NM_013362.3,NM_020198.2,NM_001220494.2,NM_001204298.1,NM_001081003.3,NM_001323563.2,NM_001386039.1,NM_001271762.1,NM_006656.5,NM_152945.3,NM_198465.4,NM_001321268.1,NM_001204056.1,NM_001288661.1,NM_001098801.2,NM_004583.4,NM_000616.4,NM_177963.4,NM_001018056.1,NM_001350474.2,NM_001142278.1,NM_001001992.2,NM_001350888.1,NM_001291927.1,NM_153339.1,NM_032104.3,NM_001127713.1,NM_001300849.1,NM_001329643.1,NM_001037540.1,NM_005808.2,NM_003595.3,NM_183075.2,NM_199165.1,NM_001324437.2,NM_000374.5,NM_001193421.2,NM_014447.4,NM_177439.3,NM_138408.3,NM_000202.8,NM_001384673.1,NM_002739.4,NM_001242919.1,NM_001322072.1,NM_002849.4,NM_001351177.2,NM_001286198.2,NM_001323916.2,NM_001387409.1,NM_001281786.1,NM_001362795.2,NM_024027.4,NM_001099666.1,NM_001080115.2,NM_033450.2,NM_001367731.1,NM_206900.3,NM_002893.3,NM_001278255.1,NM_001077527.1,NM_001300772.2,NM_001184751.1,NM_001161529.1,NM_001105520.2,NM_001100622.3,NM_005652.3,NM_013999.4,NM_001324128.1,NM_001145365.1,NM_003064.2,NM_001197238.2,NM_001385870.1,NM_033081.2,NM_024899.2,NM_001295.3,NM_001360.2,NM_001145774.1,NM_005345.6,NM_015722.3,NM_176796.3,NM_201402.2,NM_001346228.1,NM_001370458.1,NM_001352517.1,NM_023933.3,NM_001098529.1,NM_005198.4,NM_001198897.1,NM_018207.2,NM_017492.3,NM_001329671.2,NM_001365310.2,NM_001323627.2,NM_016152.4,NM_021723.5,NM_012476.3,NM_022493.1,NM_001323418.1,NM_198988.2,NM_001135585.2,NM_001100412.1,NM_001320774.1,NM_001323266.2,NM_001291219.1,NM_144571.2,NM_001330611.3,NM_001278240.2,NM_001002916.4,NM_001194986.1,NM_001387468.1,NM_152869.4,NM_001278266.2,NM_001369817.1,NM_018103.4,NM_012337.3,NM_033158.5,NM_001283026.1,NM_001376747.1,NM_006135.2,NM_001301171.1,NM_181617.1,NM_001323584.1,NM_001353692.1,NM_001375630.1,NM_015559.2,NM_001163692.1,NM_001350512.2,NM_017629.4,NM_001010887.2,NM_017527.4,NM_001253775.2,NM_001012756.2,NM_153355.5,NM_015204.2,NM_198275.2,NM_001317211.2,NM_170664.3,NM_001351369.1,NM_001330561.1,NM_001033047.2,NM_005575.3,NM_022913.3,NM_201517.2,NM_001370442.1,NM_003172.4,NM_001099681.2,NM_001385384.1,NM_001346230.2,NM_001127507.3,NM_001346105.2,NM_001376023.1,NM_003931.2,NM_178469.3,NM_001242607.2,NM_181351.5,NM_001326559.2,NM_001243247.2,NM_001330718.1,NM_198485.4,NM_001352032.2,NM_001127695.1,NM_001278126.1,NM_001015000.2,NM_001024960.3,NM_001145195.1,NM_001252676.2,NM_020205.2,NM_181272.2,NM_001384659.1,NM_001100393.3,NM_007343.3,NM_001318785.2,NM_001300984.2,NM_001254952.1,NM_033481.3,NM_003718.4,NM_001166221.2,NM_001372027.1,NM_138638.4,NM_144976.3,NM_001289038.1,NM_148169.1,NM_001353026.1,NM_181985.4,NM_001285523.3,NM_003182.2,NM_001111019.2,NM_003948.5,NM_001369588.3,NM_003881.2,NM_002499.3,NM_001146206.1,NM_001280790.2,NM_001163424.1,NM_013232.4,NM_021927.3,NM_002190.3,NM_001282456.4,NM_001254757.2,NM_001145849.1,NM_152367.3,NM_001378070.1,NM_138277.2,NM_000688.5,NM_001256712.2,NM_001142402.1,NM_001190919.1,NM_001207066.1,NM_001350438.1,NM_173591.5,NM_014643.3,NM_018899.6,NM_001605.2,NM_001792.5,NM_001198800.3,NM_032425.4,NM_020459.1,NM_001330320.1,NM_001349261.1,NM_178504.4,NM_152262.3,NM_173662.4,NM_001100599.1,NM_032587.4,NM_080549.3,NM_025015.2,NM_001281459.2,NM_001329929.1,NM_024764.2,NM_001258311.2,NM_001320046.2,NM_000384.2,NM_003174.3,NM_018913.2,NM_001320781.2,NM_001294343.2,NM_001353107.1,NM_001387059.1,NM_001304363.1,NM_014932.3,NM_001135771.2,NM_001127705.2,NM_015488.4,NM_031297.4,NM_001323571.1,NM_033033.4,NM_001127387.1,NM_001199898.1,NM_001369793.1,NM_052841.4,NM_001349155.1,NM_005438.3,NM_013352.3,NM_199001.4,NM_001257964.1,NM_004851.3,NM_013987.2,NM_001080437.3,NM_001321573.2,NM_001269040.2,NM_001381.5,NM_002685.3,NM_014884.4,NM_174941.6,NM_001388247.1,NM_001142564.2,NM_001039182.3,NM_001134440.1,NM_001170796.1,NM_012466.4,NM_001387216.1,NM_001005328.1,NM_006369.4,NM_001346550.1,NM_001199098.1,NM_201277.1,NM_001349109.1,NM_001387300.1,NM_004627.5,NM_133259.3,NM_001324215.1,NM_001300837.1,NM_002528.7,NM_030622.8,NM_001350300.2,NM_020782.1,NM_001270437.2,NM_001281534.1,NM_198332.2,NM_015665.6,NM_001363051.1,NM_058206.3,NM_001351341.1,NM_001270892.2,NM_001364827.1,NM_001271937.2,NM_001099223.2,NM_001386766.1,NM_001320709.2,NM_207012.4,NM_001164379.2,NM_016300.4,NM_001349341.1,NM_004934.4,NM_001017371.4,NM_001282231.1,NM_001130714.2,NM_020857.2,NM_015256.4,NM_001199551.1,NM_001172087.1,NM_020713.2,NM_001166254.1,NM_005999.3,NM_001195304.1,NM_001256304.3,NM_001330433.2,NM_001386171.1,NM_005310.5,NM_001077415.3,NM_001349428.1,NM_199353.2,NM_007277.4,NM_001350735.2,NM_033127.4,NM_001308382.2,NM_001349971.2,NM_033259.2,NM_145010.3,NM_001387311.1,NM_001353756.1,NM_001347023.2,NM_001322286.1,NM_001351072.1,NM_001319052.1,NM_001321941.2,NM_002773.5,NM_001005336.1,NM_001242909.1,NM_145658.4,NM_153610.5,NM_001367327.1,NM_001355243.1,NM_001127893.2,NM_001289069.1,NM_001286216.1,NM_000330.3,NM_001261427.3,NM_001005279.2,NM_000218.3,NM_001366726.1,NM_032372.5,NM_001323316.1,NM_024415.2,NM_031889.2,NM_001290252.2,NM_007136.4,NM_003630.2,NM_001282352.2,NM_001146278.2,NM_001291968.2,NM_145733.3,NM_138379.3,NM_001135731.2,NM_001165938.2,NM_001270729.2,NM_001318766.1,NM_018095.6,NM_032735.3,NM_001363296.1,NM_001170753.2,NM_144626.3,NM_005188.3,NM_005655.2,NM_001347096.1,NM_001324468.1,NM_001377947.1,NM_005902.3,NM_006165.3,NM_001242463.2,NM_001384333.1,NM_002290.3,NM_001350326.2,NM_001353566.1,NM_001367419.2,NM_001386463.1,NM_001172420.1,NM_001018075.1,NM_018253.4,NM_001304808.2,NM_145650.3,NM_001351994.1,NM_001184999.1,NM_001329676.2,NM_006925.5,NM_031442.4,NM_001177970.1,NM_001308383.1,NM_005032.7,NM_001184970.2,NM_001348071.2,NM_001289003.1,NM_005270.5,NM_144567.5,NM_152304.3,NM_001146257.1,NM_002021.3,NM_000479.3,NM_001278185.1,NM_002719.3,NM_198147.3,NM_130398.3,NM_138453.2,NM_001376758.1,NM_201265.2,NM_001322185.2,NM_017504.2,NM_173540.2,NM_001145852.2,NM_017510.4,NM_022116.7,NM_001347725.2,NM_172106.3,NM_001321149.1,NM_001316968.1,NM_001388138.1,NM_001352165.2,NM_001300858.1,NM_001122608.3,NM_001283038.1,NM_001100430.2,NM_004590.3,NM_032438.3,NM_020902.2,NM_001346458.2,NM_001008710.1,NM_001276320.2,NM_004367.5'

In [31]: client = Client()
WARNING:eutils._internal.queryservice:No NCBI API key provided; throttling to 3 requests/second; see https://ncbiinsights.ncbi.nlm.nih.gov/2017/11/02/new-api-keys-for-the-e-utilities/

In [32]: response = client.efetch('nuccore', tx_accessions_ids)

In [33]: len(tx_for_test)
Out[33]: 500

In [34]: len(list(response))
Out[34]: 250

It makes sense that there is an upper limit to the number of IDs that can be passed in to a query, but the client should propagate a warning or raise an error when a bulk query exceeds this limit.

github-actions[bot] commented 11 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions[bot] commented 10 months ago

This issue was closed because it has been stalled for 7 days with no activity.