Packages for CKAN 2.9

[amercader]

Now that we have a branch for 2.9 we can start building packages for it. I think for 2.9 we need to provide both py2 and py3 packages. My suggestion would be to target this version in this order:

1. Ubuntu 18.04 (bionic) - py2. This should be the easier to get done, as our scripts already build 2.8 on bionic
2. Ubuntu 20.04 (focal) - py3. I'd start with py3 as it's the default version included in 20.04
3. Ubuntu 20.04 (focal) - py2. I think a 20.04 py2 based install is important while the ecosystem of extensions catches up and to allow for a more gradual migration, but we should see how far we get with the other two versions first.

On all 2.9 packages we should ditch Apache and use uwsgi instead, based on https://github.com/ckan/ckan/pull/5406

Piling more stuff on top of package.yml might makes things complicated so I'm happy to start refactoring the scripts to make them more modular and separate the stuff that's needed for each version into its own module.

@kowh-ai how does this sound?

[kowh-ai]

yes @amercader the 3 CKAN 2.9 environment packages you specify look fine. If you could please start off the separation that would be great - I guess the Python version could be another variable passed into ckan-package, then a conditional could be set up in the Vagrantfile file for ansible.playbook to reference seperate package.yml files, one for py2 and one for py3…You would have a better idea though

We should get rid of all references to ‘trusty’ too :-)

You can assign this to me

[kowh-ai]

Any thoughts on the approach for CKAN Packaging to me more modular @amercader ? Perhaps ansible “include_tasks” would work….it’s simple enough - seperate yml files for related tasks, one for each version - could still have one playbook file (package.yml)

- name: Install Packages - 2.9 include_tasks: “2.9/packages.yml" when: version[0] == "2.9"

- name: Install Packages - 2.8 include_tasks: “2.8/packages.yml" when: version[0] == “2.8”

The Python version may need to be another argument to ckan-package

[kowh-ai]

@amercader - I built a CKAN 2.9 (Python 3) package for Ubuntu 20.04 today and it seemed to install OK. Iv'e also tested a CKAN 2.9. (Python 2) package for Ubuntu 18.04 on the weekend too - I basically just hacked the scripts for each release :-) - Did you want me to start off a PR for CKAN Packaging? for all versions? I'm still unclear on the best approach to make the ansible yml files more modular...The way I see it, CKAN 2.9 on Ubuntu 20.04 needs Python2 and Python3 packages which slightly complicates things.

[amercader]

@kowh-ai sorry it took a while to get back to this

Let's focus on get the packages working first and then we'll clean up the scripts, so for now we can start a completely new ansible file for the new combinations if that makes things easier (eg CKAN 2.9 on 20.04 (or one for 2.9+py2 and one for 2.9+py3))

Did the CKAN 2.9 on Ubuntu 18.04 required many changes or can we reuse the current package.yml file? I guess the replacement of Apache by uwsgi involved some changes.

Let's create a PR with the new files so it's easier to discuss and for me to test locally

[kowh-ai]

@amercader - no problem, you are very busy - I have been working on a PR yesterday and should have something by the end of today

[kowh-ai]

Draft PR: https://github.com/ckan/ckan-packaging/pull/18

[amercader]

@kowh-ai here's my update. I can build all target versions using your scripts in #19. I pushed some minor changes and haven't merged it yet, as I think we can keep working on that branch while we work on this release packages and merge afterwards.

I'm going through the process of updating the package install docs for 2.9 as there are still old unrelated things listed (eg apache). For this I'm using the 2.9 focal py3 package. Once it's done we can review that they also work for 2.9 focal py2 package, in case there is something py2 specific.

Eventually it would be good to do a quick test on as many of these as we can so if we can do some testing on some of these that would be great:

CKAN version	Target	Python version	Build	Tested
2.9	focal	py3	:heavy_check_mark:	:heavy_check_mark:
2.9	focal	py2	:heavy_check_mark:	:heavy_check_mark:
2.9	bionic	py2	:heavy_check_mark:	:heavy_check_mark:
2.8	bionic	py2	:heavy_check_mark:	:heavy_check_mark:
2.8	xenial	py2	:heavy_check_mark:	:heavy_check_mark:
2.7	xenial	py2	:heavy_check_mark:	:heavy_check_mark:

(tested successfully apart from the DataStore which is still to test)

I think we can count a version as tested if you can do all the following without errors (which should take 5 minutes once CKAN is installed):

• Install it and get CKAN running
• Create a sysadmin via CLI and log in
• Create an organization, upload an image
• Create a dataset with a CSV resource
• Check that a view was created and data is on the DataStore (green button)

[kowh-ai]

@amercader - excellent!! - I will start going through the testing on the weekend and will update here...thanks for your help!

[kowh-ai]

@amercader - so far both 2.9 focal installs (py3, py2) are fine apart from the last test: "Check that a view was created and data is on the DataStore (green button)" I get this: This resource view is not available at the moment. Could not load view: DataProxy returned an error (Request Error: Backend did not respond after 10 seconds) …The Datastore extension seems to be set up correctly…I will keep plugging away…any hints?

[amercader]

@kowh-ai that's the DataPusher not working. Its logs are on /var/log/apache/datapusher.error.log or similar. There will be clues there

[amercader]

As of now, these are the pending things I've identified:

• [ ] Docs review. Ongoing in https://github.com/ckan/ckan/pull/5511
• [ ] DataPusher is not running. In <= 2.8 it was set up in Apache, but on 2.9 we need to serve it via uwsgi as well. So we need another supervisor process + wsgi ini file + datapusher wsgi.py file (there's one in the datapusher source IIRC)
• [x] Create another supervisor process for the CKAN worker
• [x] Move app logs to /var/log/ckan (create dir, change perms, change paths in uwsgi ini file)
• [x] Set up log rotation (backed in uwsgi? or logrotate)
• [ ] Permissions issue on i18n files

[amercader]

@kowh-ai I pushed the final release versions of 2.8 and 2.7 (tags ckan-2.8.5 and ckan-2.7.8) in case you can build and test them. I'll take care of the above points and ask for help if I get stuck. On this:

Permissions issue on i18n files

Can you give more details as to when it happens?

[kowh-ai]

Regarding i18n/ directory and file creation underneath, when I'm running the ckan -c /etc/ckan/default/ckan.ini db init command as a non-root user (brett)(after . /usr/lib/ckan/default/bin/activate) I don't have write permission on this directory as it's owned by www-data:www-data. Not sure what the best option is here....I could sudo to www-data I guess...

[kowh-ai]

I pushed the final release versions of 2.8 and 2.7 (tags ckan-2.8.5 and ckan-2.7.8) in case you can build and test them.

@amercader - Yes I will test tomorrow morning

[kowh-ai]

@amercader - the tag ckan-2.8.5 breaks the regex I set up to pull out the CKAN version number eg: 2.8 I had set it up specifically to handle tags/branches like dev-v2.9...

I’m wondering the best way forward here

[amercader]

@kowh-ai \D+\D+(?P<capture>\d+\.\d+) seems to do the trick:

I highly recommend https://regex101.com/

[kowh-ai]

haha...yep Iv'e already build the new packages with this exact regex also...I'm just wondering the best approach for a more permanent solution...not urgent

And yes I used regex101 when I designed the first one. I wish this was around when I was using Perl back in the 90's LOL

[kowh-ai]

@amercader - ckan tags have been tested (everything but not including DataPusher)

• ckan-2.8.5 has been tested on xenial and bionic
• ckan-2.7.8 has been tested on xenial