Export to CKAN

[ericbusboom]

Implement a connection to CKAN, http://ckan.org, to push Ambry packages to CKAN, based on the access and source metadata.

[ This issue isn't really detailed enough to complete the task, so give it a best shot and we'll deal with details later. ]

CKAN has these fundamental organizational concepts:

• User
• Organization. A user may belong to multiple organizations.
• Dataset
• Resources

Ambry bundles are mapped to CKAN datasets. Ambry partitions are mapped to resources. When a bundle is published to CKAN:

• The CKAN documentation is built from the Ambry documentation file ( documentation.md ) along with additional documentation that is constructed automatically from meta data and appended to the CKAN documentation.
• Some of CKAN metadata is loaded from Ambry metadata, such as author and maintainer. Other Ambry metadata will be loaded into custom CKAN metadata fields.
• Partitions are extracted to CSV, KML or other formats and loaded to CKAN.
• External documentation is stored as resources, with links to the original documents.
• Write the bundle schema file to a resource as schema.csv.

There should be some additional Ambry configuration for uploading to CKAN:

• The user's default CKAN organization
• User's CKAN credentials ( in the ambry-accounts.yaml )

Don't publish bundles for these access values:

• internal
• test
• controlled
• restricted
• census

Publish publicly, with no access restrictions. The organization is based on the identity.source. :

• public

Publish to any logged in CKAN user. The organization is based on the identiy.source. :

• registered

Publish only to the organization of the user who owns the data library that the publications are being made from. The organization is the organization configured by the user. :

• private
• licensed

  Implementation Notes

It would probably be easiest to install CKAN using docker: http://extensions.ckan.org/extension/datacats/

[nmb10]

Done. Requires CKAN setup as described here - https://github.com/CivicKnowledge/ambry/tree/develop#ckan-export

Found problems:

1. API released on http://demo.ckan.org does not delete datasets - it marks them as deleted. So publishing dataset after deleting fails with 'Url already in use' error. Newer versions of the API have dataset_purge method for complete deleting.
2. Couldn't make user_role_bulk_update to work. Now it's individual role update for each user of organization.

Exporting dataset:

ambry ckan_export <dataset_vid>

[ericbusboom]

On Nov 4, 2015, at 6:11 AM, Kazbek notifications@github.com wrote:

Done. Requires CKAN setup as described here - https://github.com/CivicKnowledge/ambry/tree/develop#ckan-export

Found problems:

1. API released on http://demo.ckan.org does not delete datasets - it marks them as deleted. So publishing dataset after deleting fails with 'Url already in use' error. Newer versions of the API have dataset_purge method for complete deleting.

   • Couldn't make user_role_bulk_update to work. Now it's individual role update for each user of organization.

Excellent, thanks. I’ll have you work on the census data packages next, but I’ll need some time to describe what to do. We’ll create a new git repo for them, since there will be a lot of them ( probably about 50 )

eric.

---

Eric Busboom (619) 363 2607
eric@busboom.org http://www.busboom.org