ckanext-udc

Developer Manual

Config Schema

[   
    // The first level.
    {
        "title": "level 1 title",
        "name": "maturity_level_1",
        "fields": [
            {
                // Internal names, used in APIs. Special characters are not allowed.
                "name": "theme",

                // The Label of this field
                "label": "Domain / Topic",

                // The short description is under the field
                "short_description": "The theme.",

                // The long description is displayed when the cursor is hovered/clicked on the `quesion mark icon`
                "long_description": "The theme or topic of the package.",

                // The type of this field, default to "text".        
                // See "Supported field types" section for all supported types.
                "type": "text"
            },
            {
                // This clones the field that predefined in CKAN.
                // See "Supported CKAN Fields" section for all supported CKAN Fields
                "ckanField": "tags",

                // The short description is under the field
                "short_description": "The tags or keywords.",

                // The long description is displayed when the cursor is hovered/clicked on the `quesion mark icon`
                "long_description": "The tags or keywords of the package.",

            },
        ]
    },
    // The second level.
    {
        "title": "level 2 title",
        "name": "maturity_level_2",
        "fields": [
            {...}
        ]
    }
]

Supported CKAN Fields

Notes: short description and long description are not available for title, license, organization_and_visibility

title (*required): Title

description: Description

tags: Tags / Keywords

license_id: License

organization_and_visibility (*required): organization dropdown

source: url to the source

version: version number

author: Author/Creator

author_email

maintainer

maintainer_email

custom_fields: Custom CKAN Fields, key/value pairs

Supported field types

text
date
datetime
time

single_select

{
   "name": "access_diff_version",
   "label": "Can different versions of the data be accessed?",
   "type": "single_select",
   // A list of available options.
   // The "value" must be a string.
   "options": [
       {"text": "N/A", "value": ""},
       {"text": "Yes", "value": "true"},
       {"text": "No", "value": "false"}
   ]
}

Requirements

If your extension works across different versions you can add the following table:

Compatibility with core CKAN versions:

CKAN version	Compatible?
2.6 and earlier	not tested
2.7	not tested
2.8	not tested
2.9	not tested
2.10.X	yes
2.11.X	yes

Suggested values:

"yes"
"not tested" - I can't think of a reason why it wouldn't work
"not yet" - there is an intention to get it working
"no"

Installation

TODO: Add any additional install steps to the list below. For example installing any non-Python dependencies or adding any required config settings.

To Install NodeJS 20:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
# Reopen terminal or following the guide in the output to add 'nvm'

# Install NodeJS 20
nvm install 20

To install ckanext-udc:

Activate your CKAN virtual environment, for example:
```
. /usr/lib/ckan/default/bin/activate
```

Clone the source and install it on the virtualenv

cd /usr/lib/ckan/default/src/
git clone https://github.com/csse-uoft/ckanext-udc.git
cd ckanext-udc
pip install -e .
pip install -r requirements.txt

Add udc udc_theme udc_import udc_import_other_portals udc_react to the ckan.plugins setting in your CKAN config file (by default the config file is located at /etc/ckan/default/ckan.ini).

If you want to use the activity plugin, put it to the top (order matters).

Adjust ckan.jobs.timeout to 36000 (10 hours)

Install UDC-React Dependencies & Build

cd /usr/lib/ckan/default/src/ckanext-udc/ckanext/udc_react/ckan-udc-react
npm install
npm run build

Init DB

 # Use your own path to the ckan.ini
 ckan -c /etc/ckan/default/ckan.ini udc initdb

Restart CKAN. For example if you've deployed CKAN with Apache on Ubuntu:
```
 sudo supervisorctl reload
```

Configure deployment server to support websocket connection

Reinstall uwsgi with SSL support

sudo apt-get install libssl-dev

source /usr/lib/ckan/default/bin/activate

# Uninstall previous version of `uwsgi` if exists
pip uninstall uwsgi

## Manually build uwsgi with SSL support
# set necessary lib paths
export CFLAGS="-I/usr/include/openssl"
# aarch64-linux-gnu folder used for ARM architecture and may be different for your env
# use [apt-file list libssl-dev] to check lib folders (apt-file should be additionally installed)
export LDFLAGS="-L/usr/lib/aarch64-linux-gnu"
# activate SSL support
export UWSGI_PROFILE_OVERRIDE=ssl=true
# build uwsgi using pip (--no-use-wheel deprecated so used --no-binary instead)
# this command will install 2.0.20 version. Version may be changed or removed. It is not mandatory
pip install -I --no-binary=:all: --no-cache-dir uwsgi

# Check SSL support
uwsgi --help | grep https

Add to /etc/ckan/default/ckan-uwsgi.ini
Remove enable-threads and threads if exists, threading is not compitable with gevent.
```
gevent          =  100 # number of coroutine
http-websockets = true
```

Update nginx config sudo nano /etc/nginx/sites-enabled/ckan

Add the following after location / {...}

location /socket.io/ {
    proxy_pass http://127.0.0.1:8080/socket.io/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $host;
}

Config settings

None at present

Run as a developer

CKAN main procress

source /usr/lib/ckan/default/bin/activate
VITE_ORIGIN=http://<your-server-ip>:5173 WERKZEUG_DEBUG_PIN=223344 uwsgi --http :5000 --gevent 1000 --http-websockets --master --wsgi-file /etc/ckan/default/wsgi.py --callable application --module wsgi:application --py-autoreload=1

React Frontend

cd /usr/lib/ckan/default/src/ckanext-udc/ckanext/udc_react/ckan-udc-react/
npm run dev -- --host 0.0.0.0

CKAN worker process

source /usr/lib/ckan/default/bin/activate
WERKZEUG_DEBUG_PIN=223344 ckan -c /etc/ckan/default/ckan.ini jobs worker

Tests

To run the tests, do:

pytest --ckan-ini=test.ini

Releasing a new version of ckanext-udc

If ckanext-udc should be available on PyPI you can follow these steps to publish a new version:

Update the version number in the setup.py file. See PEP 440 for how to choose version numbers.
Make sure you have the latest version of necessary packages:

pip install --upgrade setuptools wheel twine
Create a source and binary distributions of the new version:

python setup.py sdist bdist_wheel && twine check dist/*

Fix any errors you get.
Upload the source distribution to PyPI:

twine upload dist/*
Commit any outstanding changes:

git commit -a git push
Tag the new release of the project on GitHub with the version number from the setup.py file. For example if the version number in setup.py is 0.0.1 then do:

git tag 0.0.1 git push --tags

License

AGPL

csse-uoft / ckanext-udc

readme