search

nih-cfde / cfde-deriva

Collaboration point for miscellaneous CFDE-deriva scripts
Other
2 stars 3 forks source link

Mitigate timeouts in NCPI export #371

Closed karlcz closed 1 year ago

karlcz commented 2 years ago

When NCPI export is attempted on large query results (many matching files), the export service reports a query timeout. The same export would succeed as a BDBag output.

This is due to the relatively slow export transformer code generating the NCPI manifest file. If we precompute an NCPI manifest table during the ETL phase when other portal tables are constructed, we can produce the same CSV output during export with the cheaper method used in BDBag exports.

karlcz commented 2 years ago

An implementation of this is in preview in the app-dev catalog 1.

karlcz commented 1 year ago

This was released already