search

cityofcapetown / Data-Portal

Docker image for City's data portal prototype
2 stars 1 forks source link

Size uploading limit #5

Open Gordonei opened 5 years ago

Gordonei commented 5 years ago

Using a command like this:

curl -v -X POST \
           -H "Content-Type: multipart/form-data"  \
           -H "Authorization: <API Key here>"  \
           -F "id=<Resource ID>" \
           -F "upload=@<large filename here>" \
           https://ckan.arbidata.com/api/3/action/resource_update

I'm able to upload ~500 MBs, but not ~1.4GB. I think I have changed the relevant settings in production.ini and our NGINX reverse proxy config - the failure is a 500, not a 400.

A workaround is to break datasets into smaller resources, but this causes unnecessary thinking about how to split the dataset up.

Gordonei commented 5 years ago

Another workaround is to go behind CKAN's back, and load the data straight into the Minio.

This should be done in two steps:

  1. Write a 10 MB sample via the CKAN API - this creates the necessary file in Minio, and also loads sample data into the datastore.
  2. Upload the file to Minio directly using the Minio/Boto client. The path in Minio should be ckan/resources/<resource-id>/<file name>.

Doing (1) means that various metadata in CKAN will be updated to reflect that new data has been loaded.