Development composition fails to start with several errors

[ajs6f]

Trying to start the development mode with docker-compose.dev.yml I get several seemingly-unrelated errors (and no startup). I include the full log run from docker compose up as a gist, but I extract below what seem to be some of the relevant parts. The CKAN container just keeps trying to restart, only to run into the same problems. The backend containers for Redis, Solr, PostgreSQL and the DataPusher come up healthy.

I hope there is an obvious mistake I am making?

I see sh: image_view: unknown operand reported over and over.

This db error occurs over and over:

ckan-docker-db-1          | 2023-11-27 18:16:10.901 UTC [73] ERROR:  relation "user" does not exist at character 555
ckan-docker-db-1          | 2023-11-27 18:16:10.901 UTC [73] STATEMENT:  SELECT "user".password AS user_password, "user".id AS user_id, "user".name AS user_name, "user".fullname AS user_fullname, "user".email AS user_email, "user".apikey AS user_apikey, "user".created AS user_created, "user".reset_key AS user_reset_key, "user".about AS user_about, "user".last_active AS user_last_active, "user".activity_streams_email_notifications AS user_activity_streams_email_notifications, "user".sysadmin AS user_sysadmin, "user".state AS user_state, "user".image_url AS user_image_url, "user".plugin_extras AS user_plugin_extras 
ckan-docker-db-1          |     FROM "user" 
ckan-docker-db-1          |     WHERE "user".name = 'default' OR "user".id = 'default' 
ckan-docker-db-1          |      LIMIT 1

This DataPusher-related error also occurs several times:

ckan-docker-ckan-dev-1    | Traceback (most recent call last):
ckan-docker-ckan-dev-1    |   File "/usr/bin/ckan", line 8, in <module>
ckan-docker-ckan-dev-1    |     sys.exit(ckan())
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
ckan-docker-ckan-dev-1    |     return self.main(*args, **kwargs)
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1054, in main
ckan-docker-ckan-dev-1    |     with self.make_context(prog_name, args, **extra) as ctx:
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 920, in make_context
ckan-docker-ckan-dev-1    |     self.parse_args(ctx, args)
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/cli/cli.py", line 121, in parse_args
ckan-docker-ckan-dev-1    |     result = super().parse_args(ctx, args)
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1613, in parse_args
ckan-docker-ckan-dev-1    |     rest = super().parse_args(ctx, args)
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 1378, in parse_args
ckan-docker-ckan-dev-1    |     value, args = param.handle_parse_result(ctx, opts, args)
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 2360, in handle_parse_result
ckan-docker-ckan-dev-1    |     value = self.process_value(ctx, value)
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/site-packages/click/core.py", line 2322, in process_value
ckan-docker-ckan-dev-1    |     value = self.callback(ctx, self, value)
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/cli/cli.py", line 131, in _init_ckan_config
ckan-docker-ckan-dev-1    |     _add_ctx_object(ctx, value)
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/cli/cli.py", line 140, in _add_ctx_object
ckan-docker-ckan-dev-1    |     ctx.obj = CtxObject(path)
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/cli/cli.py", line 57, in __init__
ckan-docker-ckan-dev-1    |     self.app = make_app(raw_config)
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/config/middleware/__init__.py", line 27, in make_app
ckan-docker-ckan-dev-1    |     load_environment(conf)
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/config/environment.py", line 69, in load_environment
ckan-docker-ckan-dev-1    |     p.load_all()
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/plugins/core.py", line 224, in load_all
ckan-docker-ckan-dev-1    |     load(*plugins)
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/plugins/core.py", line 255, in load
ckan-docker-ckan-dev-1    |     plugins_update()
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/plugins/core.py", line 212, in plugins_update
ckan-docker-ckan-dev-1    |     environment.update_config()
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckan/config/environment.py", line 227, in update_config
ckan-docker-ckan-dev-1    |     plugin.configure(config)
ckan-docker-ckan-dev-1    |   File "/srv/app/src/ckan/ckanext/datapusher/plugin.py", line 52, in configure
ckan-docker-ckan-dev-1    |     raise Exception(
ckan-docker-ckan-dev-1    | Exception: Config option `ckan.datapusher.api_token` must be set to use the DataPusher.
ckan-docker-ckan-dev-1 exited with code 1

and this error, which seems to be connected with db or maybe DataPusher access?

ckan-docker-ckan-dev-1    | [prerun] Initializing or upgrading db - start
ckan-docker-ckan-dev-1    | Traceback (most recent call last):
ckan-docker-ckan-dev-1    |   File "/srv/app/prerun.py", line 102, in init_db
ckan-docker-ckan-dev-1    |     subprocess.check_output(db_command, stderr=subprocess.STDOUT)
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
ckan-docker-ckan-dev-1    |     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
ckan-docker-ckan-dev-1    |   File "/usr/lib/python3.10/subprocess.py", line 526, in run
ckan-docker-ckan-dev-1    |     raise CalledProcessError(retcode, process.args,
ckan-docker-ckan-dev-1    | subprocess.CalledProcessError: Command '['ckan', '-c', '/srv/app/ckan.ini', 'db', 'init']' returned non-zero exit status 1.
ckan-docker-ckan-dev-1    | 
ckan-docker-ckan-dev-1    | During handling of the above exception, another exception occurred:
ckan-docker-ckan-dev-1    | 
ckan-docker-ckan-dev-1    | Traceback (most recent call last):
ckan-docker-ckan-dev-1    |   File "/srv/app/prerun.py", line 215, in <module>
ckan-docker-ckan-dev-1    |     init_db()
ckan-docker-ckan-dev-1    |   File "/srv/app/prerun.py", line 105, in init_db
ckan-docker-ckan-dev-1    |     if "OperationalError" in e.output:
ckan-docker-ckan-dev-1    | TypeError: a bytes-like object is required, not 'str'
ckan-docker-ckan-dev-1    | /srv/app/start_ckan_development.sh: Running init file /docker-entrypoint.d/01_setup_datapusher.sh
ckan-docker-ckan-dev-1    | Not configuring DataPusher

[iperezx]

I encountered this issue on Windows.

Are you on Windows?

If so, there might be an issue with CRLF.

On OSX (Intel), the datapusher api token is auto created but for windows, I had to add it to the .env file via one that was generated by the one on Mac:

CKAN__DATAPUSHER__API_TOKEN=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI4eURKNnlScjRCaWdGMEpKSWN0LVdncm9tZTZhajZRd01Vc0VreTZpQ01NIiwiaWF0IjoxNzAxMTIzNzE3fQ.sjOgu7hN9x8bBytB6yDIXpit4ud2ra9IpicPd3grPnM

Not sure why yet, but it allowed me to bring up all the svs.

[ajs6f]

No, this is RHEL, but thank you and I will check to see whether a manually-generated token helps.

[ajs6f]

I note that according to the following logs from the db container, the necessary schema should be in place and there seems to be no reason the query errors should be occurring...

ckan-docker-db-1          | 
ckan-docker-db-1          | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/10_create_ckandb.sh
ckan-docker-db-1          | CREATE ROLE
ckan-docker-db-1          | CREATE DATABASE
ckan-docker-db-1          | 
ckan-docker-db-1          | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/20_create_datastore.sh
ckan-docker-db-1          | CREATE ROLE
ckan-docker-db-1          | CREATE DATABASE
ckan-docker-db-1          | 
ckan-docker-db-1          | /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/30_setup_test_databases.sh
ckan-docker-db-1          | CREATE DATABASE
ckan-docker-db-1          | CREATE DATABASE
ckan-docker-db-1          | 

[ajs6f]

Trying to chase this out and following the tip @iperezx just gave me, I find in the logs:

ckan-docker-ckan-dev-1    | /srv/app/start_ckan_development.sh: Running init file /docker-entrypoint.d/01_setup_datapusher.sh
ckan-docker-ckan-dev-1    | sh: image_view: unknown operand
ckan-docker-ckan-dev-1    | Not configuring DataPusher

and in that script I see:

[sorokaa@dt-vmckantest ckan-docker]$ more ckan/docker-entrypoint.d/01_setup_datapusher.sh 
#!/bin/bash

if [[ $CKAN__PLUGINS == *"datapusher"* ]]; then
   # Datapusher settings have been configured in the .env file
   # Set API token if necessary
   if [ -z "$CKAN__DATAPUSHER__API_TOKEN" ] ; then
      echo "Set up ckan.datapusher.api_token in the CKAN config file"
      ckan config-tool $CKAN_INI "ckan.datapusher.api_token=$(ckan -c $CKAN_INI user token add ckan_admin datapusher | tail -n 1 | tr -d '\t')"
   fi
else
   echo "Not configuring DataPusher"
fi

but

[sorokaa@dt-vmckantest ckan-docker]$ docker compose config | grep PLUGINS
      CKAN__PLUGINS: envvars image_view text_view recline_view datastore datapusher

and the Docker log:

ckan-docker-ckan-dev-1    | Loading the following plugins: envvars image_view text_view recline_view datastore datapusher

clearly shows that the appropriate env var contains the appropriate values. It's also not clear to me how image_view is getting interpreted as an operand by the shell... Has anyone seen anything like this?

[ajs6f]

I've changed the test in the 01_setup_datapusher.sh to [[ $CKAN__PLUGINS =~ "datapusher" ]] and now I get a similar result to that described in https://github.com/ckan/ckan-docker/issues/101. (Along with the db-related errors and so forth reported above.) It does seem that there is some kind of weakness in the code that sets up the connection between CKAN and the DataPusher?

[GAO8A]

Getting same issue with Ubuntu 22.04.2 LTS

[iperezx]

Now getting it on OSX(Intel).

[ajs6f]

Hi @iperezx and @GAO8A,

Do you see the same

ckan-docker-ckan-dev-1    | /srv/app/start_ckan_development.sh: Running init file /docker-entrypoint.d/01_setup_datapusher.sh
ckan-docker-ckan-dev-1    | sh: image_view: unknown operand
ckan-docker-ckan-dev-1    | Not configuring DataPusher

section in the Docker logs and can you verify that the CKAN__PLUGINS env var is set as we would expect?

[iperezx]

Yes. Same .env as what I was using before.

[iperezx]

It seems like Datapusher is not ready when ckan-dev is running the default entrypoint (https://github.com/ckan/ckan-docker-base/blob/main/ckan-2.10/dev/Dockerfile#L21). If I override the entrypoint in the docker-compose file:

  ckan-dev:
    build:
      context: ckan/
      dockerfile: Dockerfile.dev
      args:
        - TZ=${TZ}
    env_file:
      - .env
    depends_on:
      db:
        condition: service_healthy
      solr:
        condition: service_healthy
      redis:
        condition: service_healthy
    ports:
      - "0.0.0.0:${CKAN_PORT_HOST}:${CKAN_PORT}"
    volumes:
      - ckan_storage:/var/lib/ckan
      - ./src:/srv/app/src_extensions
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "wget", "-qO", "/dev/null", "http://localhost:5000"]
    entrypoint: ["tail", "-f", "/dev/null"]

Then attached to the running container:

docker exec -it ckan-docker-ckan-dev-1 bash

Wait until ckan-docker-datapusher-1 outputs in the logs

ckan-docker-datapusher-1  | [pid: 9|app: 0|req: 1/1] 127.0.0.1 () {28 vars in 293 bytes} [Tue Nov 28 22:49:06 2023] GET / => generated 90 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 71 bytes (1 switches on core 0)

Then run the default entrypoint script /srv/app/start_ckan_development.sh

It runs as expected.

Trying to figure out why its an issue with the new images. Waiting on datapusher to be in a healthy state (via docker-compose depends_on) doesnt work.

[ajs6f]

I can confirm that @iperezx's commit in ckan/ckan-docker-base#38 enables the CKAN container to start for me.

[iperezx]

An alternative to not build from ckan-docker-base, one can also change the entrypoint to the following for ckan-dev:

entrypoint: ["bash","/srv/app/start_ckan_development.sh"]

[GAO8A]

@iperezx Got my dev environment to work as well, Thanks!