add lookup table module

[jqnatividad]

luau had lookup table functionality that allowed it to fetch lookup tables from CKAN, from dathere://, from a URL, and from the filesystem with some simple caching functionality. Extract lookup table logic into its own module so we can use it in other qsv commands.

[tino097]

This will work only for tables or we can use it to get the files when will be uploaded with DP+ ?

[jqnatividad]

Right now, it will only work with CSV files @tino097, but we can always extend it.

What did you have in mind?

[tino097]

Updating the fucntionality for getting the resources in DP+. Right now we are doing request download but if the resource is in the same location as the qsv it could be much faster to perform the action of pushing the data to the datastore

[jqnatividad]

Great idea! I've been thinking about a get command that fetches resources more efficiently:

• has configurable schemes (like dathere://), so we can have it fetch from different sources (e.g. S3, minio, cloud storage, etc.)
• with more sophisticated caching - building on the simple cache aging in lookup tables (using ETags, etc.)
• auto compress/decompression in a local cache

Will add it to the backlog.