add S3 compatible remote storage + add CI for populating the storage

whiletrue-industries / odds

ckangpt

MIT License

1 stars 0 forks source link

Closed OriHoch closed 1 year ago

OriHoch commented 1 year ago

added S3-compatible remote storage using Wasabi
added backend scrape-ckan-instances command which scrapes the ckan instances and saves the data to the remote storage
modified backend describe-datasets to save the dataset descriptions to the remote storage
added manual github actions for running the commands

OriHoch commented 1 year ago

@akariv @noamoss please review

I'm merging because I want to test the github actions, but please review..