datopian / metastore-lib

🗄️ Library for storing dataset metadata, with versioning support and pluggable backends including GitHub.
https://tech.datopian.com/versioning/
MIT License
10 stars 1 forks source link

GitHubStorage: Create LFS config and pointer files #5

Closed shevron closed 4 years ago

shevron commented 4 years ago

When the lfs_server_url config option is set, we should start creating and committing LFS pointer files and LFS config based on resources we get:

pdelboca commented 4 years ago

In this ticket we discussed how to implement the scenario in which two resources has the same name (possible in CKAN) given that git doesn't allow to filenames with the same name in a folder.

Basically:

shevron commented 4 years ago

We are not going to handle this in metastore-lib, but just expect path values for resources to be set correctly in a non-conflicting way by the calling code.

In the GitHub storage backend, we will do some last minute sanity check on resource paths: before committing them:

When integrating with CKAN, path normalization and de-duplication will be handled in ckanext-versioning or a conversion library used by it. See https://github.com/datopian/ckanext-versioning/issues/2