Now that qsv has a separate lookup module (#2262), create a get command that can retrieve tabular data from various sources:
local filesystem
URL (http, https, sftp and dathere schemes supported)
dathere:// is a special scheme that autoresolves to either a datHere-provisioned CKAN or a datHere host, based on the datHere URL
The command can be used to efficiently retrieve reference data; to seed the caches for luau (lookup tables), validate JSONschema dynamicEnum validation; and to have a more efficient resource fetcher for DataPusher+ (should greatly improve harvesting throughput).
Now that qsv has a separate lookup module (#2262), create a
getcommand that can retrieve tabular data from various sources:dathere://is a special scheme that autoresolves to either a datHere-provisioned CKAN or a datHere host, based on the datHere URLIt will be backed by a diskcache using https://crates.io/crates/cached (already heavily used by
fetchandgeocodecommands) and will use https://docs.rs/http-cache-reqwest/latest/http_cache_reqwest/ as well to make it even more efficient in fetching remote data (using http caching semantics, e.g. etag, cache keywords in response header, etc.)The command can be used to efficiently retrieve reference data; to seed the caches for
luau(lookup tables),validateJSONschemadynamicEnumvalidation; and to have a more efficient resource fetcher for DataPusher+ (should greatly improve harvesting throughput).cc @tino097