search

opendata-swiss / ckanext-switzerland-ng

GNU Affero General Public License v3.0
6 stars 4 forks source link

Make extension CKAN 2.9.5 and Python 3 compatible #209

Open gavram opened 2 years ago

gavram commented 2 years ago

In order to make this extension to be compatible with CKAN 2.9.x and Python 3 some changes have to be done. Here is the list of changes:

ckanext-dcatapchharvest

  1. https://github.com/opendata-swiss/ckanext-dcatapchharvest/blob/master/requirements.txt#L1 - change iribaker==0.1.2 to iribaker==0.2
  2. https://github.com/opendata-swiss/ckanext-dcatapchharvest/blob/master/ckanext/dcatapchharvest/dcat_helpers.py#L3 - replace 
    from urlparse import urlparse

    with

    try:
from urlparse import urlparse
except:
from urllib.parse import urlparse

    to be compatible with Py2 and Py3, both

ckanext-switzerland-ng

  1. https://github.com/opendata-swiss/ckanext-switzerland-ng/blob/master/ckanext/switzerland/plugins.py#L3, replace from ckan.common import OrderedDict with from collections import OrderedDict
  2. https://github.com/opendata-swiss/ckanext-switzerland-ng/blob/master/ckanext/switzerland/helpers/backend_helpers.py#L10, replace 
    from urlparse import urlparse

    with

    try:
from urlparse import urlparse
except:
from urllib.parse import urlparse
  3. https://github.com/opendata-swiss/ckanext-switzerland-ng/blob/master/ckanext/switzerland/helpers/format_utils.py#L7 - the same as above
  4. In https://github.com/opendata-swiss/ckanext-switzerland-ng/blob/master/ckanext/switzerland/helpers/format_utils.py#L26 replace iteritems() with items()
  5. In addition, using webassets have to be implemented in templates

Here is the custom Dockerfile I'm using for the setup:

FROM debian:10

# Setting the locale   
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y locales
RUN DEBIAN_FRONTEND=noninteractive locale-gen de_DE.UTF-8

# Internals, you probably don't need to change these
ENV APP_DIR=/srv/app
ENV SRC_DIR=/srv/app/src
ENV CKAN_INI=${APP_DIR}/production.ini
ENV PIP_SRC=${SRC_DIR}
ENV CKAN_STORAGE_PATH=/var/lib/ckan
ENV GIT_URL=https://github.com/ckan/ckan.git
# CKAN version to build
ENV GIT_BRANCH=ckan-2.9.4
# Customize these on the .env file if needed
ENV CKAN_SITE_URL=http://localhost:5000

WORKDIR ${APP_DIR}

# Install required system packages
RUN apt-get -q -y update \
    && DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade

RUN apt-get -q -y install \
    python3 \
    python3-dev \
    python3-pip \
    virtualenv \
    python3-wheel \
    libpq-dev \
    libxml2-dev \
    libxslt-dev \
    libgeos-dev \
    libssl-dev \
    libffi-dev \
    libsasl2-dev  \
    libldap2-dev \
    libssl-dev \
    postgresql-client \
    build-essential \
    git-core \
    wget \
    curl \
    zlib1g-dev \
    libpcre3 \
    libpcre3-dev \
    sudo \
    uwsgi \
    uwsgi-dev \
    uwsgi-plugin-python3 \
    uwsgi-extra

RUN apt-get -q -y install \
    python3-venv

RUN apt-get -q clean \
    && rm -rf /var/lib/apt/lists/*

# Create SRC_DIR
RUN mkdir -p ${SRC_DIR} && \
    # Install pip and supervisord
    curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/get-pip.py && \
    python3 ${SRC_DIR}/get-pip.py && \
    pip3 install supervisor && \
    mkdir /etc/supervisord.d && \
    #pip wheel --wheel-dir=/wheels uwsgi gevent && \
    rm -rf ${SRC_DIR}/get-pip.py

# set python to point to python3
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10

COPY common/supervisord.conf /etc

# Define environment variables
ENV CKAN_HOME /usr/lib/ckan
ENV CKAN_VENV $CKAN_HOME/venv
ENV CKAN_CONFIG /etc/ckan

# Setup virtual environment for CKAN
RUN mkdir -p $CKAN_VENV $CKAN_CONFIG $CKAN_STORAGE_PATH && \
    python3 -m venv $CKAN_VENV && \
    ln -s $CKAN_VENV/bin/pip3 /usr/local/bin/ckan-pip3 &&\
    ln -s $CKAN_VENV/bin/ckan /usr/local/bin/ckan

ENV PATH=${CKAN_VENV}/bin:${PATH}

# Clone CKAN
RUN git clone -b ${GIT_BRANCH} ${GIT_URL} $CKAN_VENV/src/ckan
RUN cp -r $CKAN_VENV/src/ckan ${SRC_DIR}/ckan

# Create a local user and group to run the app
RUN groupadd -g 92 -r ckan
RUN adduser --uid 92 --home /srv/app --no-create-home --disabled-password --system --ingroup ckan --ingroup sudo ckan

# Install CKAN in virtual environment
RUN ckan-pip3 install -U pip && \
    ckan-pip3 install --upgrade --no-cache-dir -r $CKAN_VENV/src/ckan/requirement-setuptools.txt && \
    ckan-pip3 install --upgrade --no-cache-dir -r $CKAN_VENV/src/ckan/requirements.txt && \
    ckan-pip3 install -e $CKAN_VENV/src/ckan/ && \
    ln -s $CKAN_VENV/src/ckan/ckan/config/who.ini $CKAN_CONFIG/who.ini && \
    cp -v $CKAN_VENV/src/ckan/contrib/docker/ckan-entrypoint.sh /ckan-entrypoint.sh && \
    chmod +x /ckan-entrypoint.sh && \
    chown -R ckan:ckan $CKAN_HOME $CKAN_VENV $CKAN_CONFIG $CKAN_STORAGE_PATH

# Install ckanext-envvars
RUN ckan-pip3 install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars

RUN ckan generate config ${CKAN_INI} && \
    ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \
    ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}" && \
    ckan config-tool ${CKAN_INI} -s logger_ckanext -e level=INFO

# Create local storage folder
RUN chown -R ckan:ckan $CKAN_STORAGE_PATH

# Create entrypoint directory for children image scripts
RUN mkdir /docker-entrypoint.d

RUN chown ckan -R ${APP_DIR}

# Set timezone
ENV TZ=UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

#### Install Extensions ####

RUN ckan-pip3 install --no-cache-dir -e git+https://github.com/datopian/ckanext-auth.git#egg=ckanext-auth && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-scheming.git#egg=ckanext-scheming && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-fluent/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-fluent.git#egg=ckanext-fluent && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-showcase/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-showcase.git#egg=ckanext-showcase && \
    ckan-pip3 install -e git+https://github.com/davidread/ckanext-hierarchy.git#egg=ckanext-hierarchy && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-hierarchy/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-harvest.git#egg=ckanext-harvest && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-harvest/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/ckan/ckanext-dcat.git#egg=ckanext-dcat && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-dcat/master/requirements.txt && \
    # ckan-pip3 install -e git+https://github.com/opendata-swiss/ckanext-dcatapchharvest.git#egg=ckanext-dcataapchharvest && \
    # ckan-pip3 install -r https://raw.githubusercontent.com/opendata-swiss/ckanext-dcatapchharvest/master/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/datopian/ckanext-dataexplorer-react.git@webassets#egg=ckanext-dataexplorer-react && \
    ckan-pip3 install -r https://raw.githubusercontent.com/datopian/ckanext-s3filestore/ckan-2.9/requirements.txt && \
    ckan-pip3 install -e git+https://github.com/datopian/ckanext-s3filestore.git@ckan-2.9-image-upload-fix#egg=ckanext-s3filestore && \
    ckan-pip3 install -e git+https://github.com/keitaroinc/ckanext-issues@ef694bdfbc223f833a9ba114a8a9ac50cc4520d1#egg=ckanext-issues
# Install ckanext-pages
RUN ckan-pip3 install -e git+https://github.com/ckan/ckanext-pages.git@v0.3.1#egg=ckanext-pages && \
    ckan-pip3 install --no-cache-dir -r ${SRC_DIR}/ckanext-pages/requirements.txt
# Install ckanext-pdfview
RUN ckan-pip3 install ckanext-pdfview

# Install Xloader
RUN ckan-pip3 install ckanext-xloader && \
    ckan-pip3 install -r https://raw.githubusercontent.com/ckan/ckanext-xloader/master/requirements.txt && \
    ckan-pip3 install -U requests[security]

# RUN ckan-pip3 install -r https://raw.githubusercontent.com/opendata-swiss/ckanext-switzerland-ng/master/requirements.txt && \
#     ckan-pip3 install -e git+https://github.com/opendata-swiss/ckanext-switzerland-ng.git#egg=ckanext-switzerland

# Install ckanext-switzerland for local development
COPY src/ckanext-switzerland /srv/app/src/ckanext-switzerland
RUN pip install -r file:///srv/app/src/ckanext-switzerland/requirements.txt && \
    pip install -e file:///srv/app/src/ckanext-switzerland#egg=ckanext-switzerland

##### Local development #####
# Clone the extensions in ckan/src folder, before you start your dev work
# Install ckanext-dcataapchharvest for local development
COPY src/ckanext-dcataapchharvest /srv/app/src/ckanext-dcataapchharvest
RUN pip install -r file:///srv/app/src/ckanext-dcataapchharvest/requirements.txt && \
    pip install -e file:///srv/app/src/ckanext-dcataapchharvest#egg=ckanext-dcataapchharvest

COPY setup/prerun.py ${CKAN_VENV}
COPY setup/supervisor.worker.conf /etc/supervisord.d/worker.conf
COPY setup/start_ckan.sh ${APP_DIR}
ADD https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/wsgi.py ${CKAN_VENV}
RUN cp $CKAN_VENV/src/ckan/ckan/config/who.ini ${APP_DIR}/who.ini

ENV CKAN__PLUGINS envvars image_view text_view webpage_view pdf_view hierarchy_display hierarchy_form datastore xloader resource_proxy auth dataexplorer_view dataexplorer_table_view dataexplorer_chart_view dataexplorer_map_view fluent dcat_ch_rdf_harvester ogdch_dcat ogdch ogdch_pkg ogdch_res ogdch_group ogdch_org ogdch_showcase ogdch_archive

RUN ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \
    ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}"

# Apply Patches
# RUN if [ -d "$APP_DIR/patches/ckan" ] ; then rm -rf $APP_DIR/patches/ckan ; fi
# COPY patches ${APP_DIR}/patches
# RUN for d in $APP_DIR/patches/*; do \
#     for f in `ls $d/*.patch | sort -g`; do \
#     cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
#     done ; \
#     done

EXPOSE 5000

HEALTHCHECK --interval=60s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1

RUN chmod +x /srv/app/start_ckan.sh

# RUN ckan config-tool ${CKAN_INI} "ckanext.dcat.rdf.profiles = euro_dcat_ap dcatap_de" && \
#     ckan config-tool ${CKAN_INI} "ckanext.dcat.clean_tags = true" && \
#     ckan config-tool ${CKAN_INI} "ckan.harvest.log_level = info"

USER ckan

CMD ["/srv/app/start_ckan.sh"]
gavram commented 2 years ago

I can create PR for all except the webassets.

anuveyatsu commented 2 years ago

Thanks @gavram 👍

Hey @stefina I'm pinging you here so that you have a view of our progress and issues with installing this extension 😃

student2017USA commented 2 years ago

Hallo have you done this? I wanted using this therms because looks great!

I need help for point 5: In addition, using webassets have to be implemented in templates https://docs.ckan.org/en/2.9/theming/webassets.html https://github.com/ckan/ckan/wiki/Python-3-migration-guide-for-extensions#web-assets-js-and-css

or when you agree, can you push the full therms to me or make this public on GitHub?

`` python --version Python 3.6.9 pip --version pip 21.3.1 from /usr/lib/ckan/default/lib/python3.6/site-packages/pip (python 3.6) lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.6 LTS Release: 18.04 Codename: bionic

``

Thanks so much for your fast help!