Alembic error when adding this extension to ckan-docker

[dcrendon]

Hello GSA Team,

I am encountering an Alembic "script_location" error while trying to add this extension to the basic ckan-docker setup. The build completes successfully, but the error occurs when running the container. Previously, I managed to get CKAN running with the Harvest and DCAT extensions, but now I am facing issues with the Datajson extension.

Any help would be appreciated. I am new to CKAN, so please let me know if there are any steps I might be missing to get this working. For a bit of background, NASA is transitioning to CKAN, and I'm hoping to leverage this extension to generate a data.json file so that catalog.data.gov can continue harvesting. Below, I have posted the error I am seeing and my current CKAN Dockerfile.

Error:

ckan-dev-1    | Traceback (most recent call last):
ckan-dev-1    |   File "/usr/bin/ckan", line 8, in <module>
ckan-dev-1    |     sys.exit(ckan())
ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
ckan-dev-1    |     return self.main(*args, **kwargs)
ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1055, in main
ckan-dev-1    |     rv = self.invoke(ctx)
ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
ckan-dev-1    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
ckan-dev-1    |     return _process_result(sub_ctx.command.invoke(sub_ctx))
ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
ckan-dev-1    |     return ctx.invoke(self.callback, **ctx.params)
ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 760, in invoke
ckan-dev-1    |     return __callback(*args, **kwargs)
ckan-dev-1    |   File "/srv/app/src/ckan/ckan/cli/db.py", line 66, in upgrade
ckan-dev-1    |     _run_migrations(plugin, version)
ckan-dev-1    |   File "/srv/app/src/ckan/ckan/cli/db.py", line 124, in _run_migrations
ckan-dev-1    |     repo.upgrade_db(version)
ckan-dev-1    |   File "/srv/app/src/ckan/ckan/model/__init__.py", line 350, in upgrade_db
ckan-dev-1    |     alembic_upgrade(self.alembic_config, version)
ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/alembic/command.py", line 302, in upgrade
ckan-dev-1    |     script = ScriptDirectory.from_config(config)
ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/alembic/script/base.py", line 156, in from_config
ckan-dev-1    |     raise util.CommandError(
ckan-dev-1    | alembic.util.exc.CommandError: No 'script_location' key found in configuration.
ckan-dev-1 exited with code 0

Current dockerfile.dev:

FROM ckan/ckan-dev:2.10

RUN pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@master#egg=ckanext-harvest'
RUN pip3 install -r ${APP_DIR}/src/ckanext-harvest/pip-requirements.txt

RUN pip3 install -e 'git+https://github.com/GSA/ckanext-datajson#egg=ckanext-datajson'
RUN pip3 install -r ${APP_DIR}/src/ckanext-datajson/requirements.txt
RUN pip3 install -r ${APP_DIR}/src/ckanext-datajson/dev-requirements.txt
RUN cd ${APP_DIR}/src/ckanext-datajson && python setup.py develop

COPY docker-entrypoint.d/* /docker-entrypoint.d/

COPY patches ${APP_DIR}/patches

RUN for d in $APP_DIR/patches/*; do \
        if [ -d $d ]; then \
            for f in `ls $d/*.patch | sort -g`; do \
                cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
            done ; \
        fi ; \
    done
`

[dcrendon]

I was able to get it working by not installing the requirements after installing the extension, not sure why that fixed it though.

Updated dokcerfile.dev:

FROM ckan/ckan-dev:2.10

### Harvester ###
RUN pip3 install -e 'git+https://github.com/ckan/ckanext-harvest.git@master#egg=ckanext-harvest'
RUN pip3 install -r ${APP_DIR}/src/ckanext-harvest/pip-requirements.txt
# will also require gather_consumer and fetch_consumer processes running (please see https://github.com/ckan/ckanext-harvest)

## DCAT ###
RUN  pip3 install -e 'git+https://github.com/ckan/ckanext-dcat.git#egg=ckanext-dcat'
RUN pip3 install -r ${APP_DIR}/src/ckanext-dcat/requirements.txt

### S3FileStore ###
RUN  pip3 install -e 'git+https://github.com/keitaroinc/ckanext-s3filestore#egg=ckanext-s3filestore'
COPY plugins/s3filestore/uploader.py ${APP_DIR}/src/ckanext-s3filestore/ckanext/s3filestore/uploader.py
COPY plugins/s3filestore/click_commands.py ${APP_DIR}/src/ckanext-s3filestore/ckanext/s3filestore/click_commands.py
RUN pip3 install -r ${APP_DIR}/src/ckanext-s3filestore/requirements.txt

### DataJson ###
RUN pip3 install -e 'git+https://github.com/GSA/ckanext-datajson#egg=ckanext-datajson'

# Clone the extension(s) your are writing for your own project in the `src` folder
# to get them mounted in this image at runtime

# Copy custom initialization scripts
COPY docker-entrypoint.d/* /docker-entrypoint.d/

# Apply any patches needed to CKAN core or any of the built extensions (not the
# runtime mounted ones)
COPY patches ${APP_DIR}/patches

RUN for d in $APP_DIR/patches/*; do \
        if [ -d $d ]; then \
            for f in `ls $d/*.patch | sort -g`; do \
                cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \
            done ; \
        fi ; \
    done

[FuhuXia]

@dcrendon

• Installing packages/versions defined in ckanext-datajson/requirements.txt is recommended. But since those packages are pretty general, most likely there are installed from other extensions or ckan core, so you might be able to get away without it.
• Installing ckanext-datajson/dev-requirements.txt is not needed, unless you are doing datajson extension development and need to test your code change.
• cd into ckanext-datajson/ and run setup is not needed since it is already installed by previous command pip3 install.
• Not sure about the Alembic error message. Using your Dockerfile steps I could not replicated it. If you can remove docker/db containers/networks/volumes before each docker run, it might help to stop Alembic glitches.
• Do pay attention to the first line in the https://github.com/GSA/ckanext-datajson/blob/main/requirements.txt#L1-L2. I just noticed it defines a old version of harvest extention. I will have it updated to the latest stable release version of harvest, or you can grab the change from this PR to see if it helps to eliminate the Alembic error.

[dcrendon]

Thank you for the response @FuhuXia! This clears up a lot, it makes sense that installing duplicate/older requirements could cause issues. Currently I'm able to get away with not installing the requirements for the datajson extension, I ran a test harvest on https://open.gsa.gov/data.json and everything seems to have ran correctly.