search

TkTech / ckanext-cloudstorage

Implements support for resource storage against multiple popular providers via apache-libcloud (S3, Azure Storage, etc...)
MIT License
35 stars 56 forks source link

resource_create API request returning 504 timeout error #39

Open hammadhasandogar opened 5 years ago

hammadhasandogar commented 5 years ago

api/action/resource_create works fine with local storage ckan.storage_path = /var/lib/ckan/default but after enabling cloudstorage it gives timeout error while making request from UI or Postman

am i missing anything here?

hammadhasandogar commented 5 years ago

I was able to resolve the timeout error, now metadata get saved but not the file, neither on S3 nor on local. After diving into code, I think below is what I got to do:

  1. Call resource_createAPI - it saves the metadata 
    curl -H'Authorization: <api key>' \
'http://<instance ip>/api/action/resource_create' \
--form upload=@metadata.sql --form package_id=public --form name="curl file"
  2. Then cloudstorage_initiate_multipart - it gives back upload key 
    curl -H'Authorization: <api key>' \
'http://<instance ip>/api/action/cloudstorage_initiate_multipart' \
-d '{"id":"e3876b08-27d2-42ba-b088-ee4789091b83","name":"metadata.sql","size":3610 }'
  3. Then cloudstorage_upload_multipart - not working, gives error(AttributeError: SpooledTemporaryFile instance has no attribute 'file') on ckanext/cloudstorage/logic/action/multipart.py", line 155, in upload_multipart 
    curl -H'Authorization: <api key>' \
'http://<instance ip>/api/action/cloudstorage_upload_multipart' \
--form "id=e3876b08-27d2-42ba-b088-ee4789091b83" --form "upload=@metadata.sql" --form "partNumber=1" --form "uploadId=XXXXXXXXX.qqXAQPCxZFij0U1pVpsbVjpzkO3H6Rtang1x2FVS.cJgWkQPjbKvCNzoPQZ2S09dsmG4FcQI_qyRWmRUz3sHWLWgh9L3lEckOAJteeaIHBvlC6tniLh1gu"
  4. Finally cloudstorage_finish_multipart - not working, gives error (Error in multipart commit: The XML you provided was not well-formed or did not validate against our published schema (MalformedXML)) on ckanext/cloudstorage/logic/action/multipart.py", line 198, in finish_multipart, because the option 3 wasn't complete i guess 
    curl -H'Authorization: <api key>' \
'http://<instance ip>/api/action/cloudstorage_finish_multipart' \
-d '{"save_action":"go-metadata","id":"e3876b08-27d2-42ba-b088-ee4789091b83","uploadId":"XXXXXXXXX.qqXAQPCxZFij0U1pVpsbVjpzkO3H6Rtang1x2FVS.cJgWkQPjbKvCNzoPQZ2S09dsmG4FcQI_qyRWmRUz3sHWLWgh9L3lEckOAJteeaIHBvlC6tniLh1gu"}'

Can anyone confirm please?

TkTech commented 5 years ago

Were you ever able to resolve this?