I am trying to prepare data for the Open Targets Platform data pipeline using the code in https://github.com/opentargets/platform-input-support. By far the slowest part of this script is the retrieval of data from ChEMBL through the REST interface, in particular from the molecule endpoint. If we could retrieve data with larger page size this script would run much faster.
Alternatively, does the ChEMBL team make individual tables available for download? I've found the ChEMBLdb download but it's over a GB so it takes quite some time to download.
How was the value
1000chosen for https://github.com/chembl/chembl_webservices_2/blob/master/chembl_webservices/core/pagination.py#L24? There does not seem to have been much discussion around this decision: https://github.com/chembl/chembl_webservices_2/issues/15.I am trying to prepare data for the Open Targets Platform data pipeline using the code in https://github.com/opentargets/platform-input-support. By far the slowest part of this script is the retrieval of data from ChEMBL through the REST interface, in particular from the
moleculeendpoint. If we could retrieve data with larger page size this script would run much faster.Alternatively, does the ChEMBL team make individual tables available for download? I've found the ChEMBLdb download but it's over a GB so it takes quite some time to download.