hydra: Manage large resources exceptions differently

[geoffreyaldebert]

Today, we manage resource_id exceptions through a config file.

I propose to manage it from a postgres table instead. So :

• create a table resource_exception in hydra-hydra with a column resource_id
• hydra will check this table to prioritize its db-ification even if it does not respect criterias (as filesize)

These kind of tables are heavy, so we want to optimize their future queries. So we have to add indexes. We can add indexes into the table tables_index in hydra-hydra-csv with a new column indexes that is empty for classical resource and a list of column names for heavy files. So :

• when a big ressource will be db-ificate, hydra will check if there is some indexes to create
• it creates indexes on this specific table

It will be very nice to add an exception through a POST query to hydra, for instance :

POST https://crawler.data.gouv.fr/api/add-exception
{
    "resource_id" : "XXXX",
    "indexes": ["COL1", "COL32", "COL43"]
}

[bolinocroustibat]

Note: this should replace config.LARGE_RESOURCES_EXCEPTIONS

[bolinocroustibat]

PR: https://github.com/datagouv/hydra/pull/148

[bolinocroustibat]

Question: would it be more performant to do a SQL query using a SQL function to check if the resource id is in the table, instead of getting them and doing a Python comparison?