Write documentation for EPUB producers on how to validate EPUBs

josteinaj commented 4 years ago

Some sort of user guide? Could be an appendix to the guidelines?

If we make it possible to validate outside of Pipeline 2 (#395), then we could have a user guide for that as well.

martinpub commented 3 years ago

I have written a pragmatic user guide for using the Nordic Mig. validator in Docker (on Windows). We discussed it in the last validator meeting and decided it can be added to the Nordic Migrator web page.

However there are some issues left:

Port forwarding/exposure to host in Docker on Windows is only working with the linking method
Restarting the web UI container after reboot is not working properly (it seems there is a PID file lingering, so it's not gracefully shut down). Having to set up the web UI after restart is not working in a production workflow.

Any help with these issues will be greatly appreciated!

martinpub commented 3 years ago

Hi @josteinaj, we forgot to discuss this in relation to #500 at today's meeting. But probably the web page needs to be updated as well.

josteinaj commented 3 years ago

Sounds great!

Port forwarding/exposure to host in Docker on Windows is only working with the linking method

I don't have much experience with Docker in Windows so I'm not much help here, sorry.

Restarting the web UI container after reboot is not working properly (it seems there is a PID file lingering, so it's not gracefully shut down). Having to set up the web UI after restart is not working in a production workflow.

When you say "restarting the container", do you mean that there's a stopped Docker container after reboot, and when you resume it, it doesn't work?

At NLB we run it as a docker service. When we kill a container, or reboot the computer, a new clean container is instantiated based on the official webui image. I've never had to deal with paused/stopped containers that are resumed/restarted.

The PID file is not always deleted for some reason, and it is annoying having to delete it. But when you're using Docker, each container should have its own PID file, and when you create a container from the webui image for the first time, there's never an existing PID file there, so it shouldn't be a problem.

In my experience it's a good idea to start a fresh webui once in a while as well. If it's been running for a long time, it can get a bit slow (I don't know why).

martinpub commented 2 years ago

Thanks (some months later) for the input @josteinaj. I will use this as input for further testing/documentation.

martinpub commented 2 years ago

Regarding the PID file problem, just wanted to note the issue was raised here as well: https://github.com/daisy/pipeline-webui/issues/137.

For the documentation and release part, I will focus on a workaround along the lines of what @josteinaj is describing.

martinpub commented 2 years ago

@josteinaj, would you be willing to share your Docker services configuration?

Currently I use the following docker-compose.yml, which works well with docker-compose up/docker-compose down.:

services:
  pipeline:
    image: nlbdev/nordic-epub3-dtbook-migrator:20210901
    environment:
      PIPELINE2_WS_HOST: "0.0.0.0"
      PIPELINE2_WS_AUTHENTICATION: "false"
    volumes:
      - "pipeline-data:/opt/daisy-pipeline2/data"
  webui:
    image: daisyorg/pipeline-webui
    environment:
      DAISY_PIPELINE2_URL: http://pipeline:8181/ws
    ports:
      - "9000:9000"
    volumes:
      - "webui-data:/opt/daisy-pipeline2-webui/data"
    depends_on:
      pipeline:
        condition: service_healthy
volumes:
  pipeline-data:
  webui-data:

Resuming after Docker restart is succesful, but not after Windows system restart. My hypothesis is that the webui container is not quick enough to respond to a sigterm signal, and therefore ungracefully shut down, which means the PID is still there. But as you say, the easiest way around this is just to start a new fresh container. I think it would be best if this could be handled at the configuration file level, keeping the command/process handling for the producers to a minimum. A combination of a "restart: always" policy and a "start with building a fresh container instead of resuming", but I'm not sure how to to define the latter in the configuration file?

josteinaj commented 2 years ago

Sure, here goes:

First, we use Docker Swarm, which uses the Docker Compose YAML syntax, but there might be some differences so I don't know if this will work out-of-the-box using Docker Compose.

We run two instances of the Web UI in production, and two instances in test. We run multiple instances and versions of the Pipeline 2 engine in production and in test. The following is just a single Web UI and a single nordic migrator engine. We also have a service that restarts the Web UI once a day, as we've experienced slowness and other problems in the Web UI when we keep it running for a long time.

As you can see, we still use version 2.7.0 of the Web UI. And version 1.4.6 of the nordic migrator. It should really just be a matter of changing the version number for those two, but I haven't tried yet.

For efficiency, I've mounted subdirectories from a network drive (/media/nlb-epub) as volumes on the Web UI and engine services, so that they appear to each other as if they're running on the same file system. This way, they can send each other references to the file system, instead of zipping and sending the whole book. We only do that for engines that are used with the Web UI. Engines that are not used with a Web UI do not use a network drive to store data.

`docker-stack.yml` + `docker-stack.prod.yml`

We separate generic configuration (common for prod/test/dev), and environment specific information into separate files. I've combined the configuration of a Web UI service and a nordic engine service into the following code.

We deploy these using docker stack deploy ….

version: "3.8"

# Note that secrets has to be versioned:
# https://github.com/moby/moby/issues/29882
# So whenever the secret file is changed, you
# need to give it a new version.
secrets:
  pipeline_webui_application_conf_v4:
    file: secret/pipeline-webui/prod.application.conf

services:
  pipeline-webui:
    image: daisyorg/pipeline-webui:v2.7.0
    entrypoint: /opt/daisy-pipeline2-webui/bin/pipeline2-webui -Dhttp.proxyHost="$proxyHost" -Dhttp.proxyPort="$proxyPort" -Dhttp.nonProxyHosts="$nonProxyHosts"
    ports:
      - "9000:9000"
    deploy:
      replicas: 1
      restart_policy:
        condition: any
    env_file:
      - secret/pipeline-webui/prod.env
    secrets:
      - source: pipeline_webui_application_conf_v4
        target: /opt/daisy-pipeline2-webui/conf/application.conf
    volumes:
      - type: bind
        source: /media/nlb-epub/daisy-pipeline-2-data/pipeline-engine-nordic
        target: /opt/daisy-pipeline2/data/jobs
      - type: bind
        source: /media/nlb-epub/daisy-pipeline-2-data/pipeline-webui
        target: /opt/daisy-pipeline2-webui/data/jobs

  pipeline-webui-restarter:
    image: docker
    deploy:
      replicas: 1
      restart_policy:
        condition: any
    volumes: ["/var/run/docker.sock:/var/run/docker.sock"]
    # NOTE: this can cause an error if you update the stack at the same time as this service triggers an update:
    #       rpc error: code = Unknown desc = update out of sequence
    #       if this is a problem, scale this service to 0 instances before retrying:
    #       docker service scale nlb_pipeline-webui-restarter=0
    command: ["/bin/sh", "-c", "while true; do sleep 20; if [ \"`date \"+%H:%M\"`\" = \"23:00\" ] ; then docker service update nlb_pipeline-webui --force=true ; else echo \"`date \"+%H:%M\"`\" != \"23:00\" ; fi ; done"]

  pipeline-engine-nordic:
    image: nlbdev/nordic-epub3-dtbook-migrator:1.4.6
    ports:
      - "18150:18150"
    deploy:
      replicas: 1
      restart_policy:
        condition: any
    env_file:
      - secret/pipeline-engine/prod.env
    environment:
      - PIPELINE2_WS_PORT=18150
      - PIPELINE2_WS_LOCALFS=true
    volumes:
      - type: bind
        source: /media/nlb-epub/daisy-pipeline-2-data/pipeline-engine-nordic
        target: /opt/daisy-pipeline2/data/jobs
      - type: bind
        source: /media/nlb-epub/daisy-pipeline-2-data/pipeline-webui
        target: /opt/daisy-pipeline2-webui/data/jobs

`secret/pipeline-webui/prod.application.conf`

(sensitive information removed, replace with your own secrets)

# Secret key
# ~~~~~
# See http://www.playframework.com/documentation/latest/ApplicationSecret for more details.
play.crypto.secret="SECRET"

# The application languages
# ~~~~~
play.i18n.langs = [ "en" ]

# Max content length
play.http.parser.maxDiskBuffer = 1G

# Database configuration
# ~~~~~
db.mysql.driver=com.mysql.cj.jdbc.Driver
db.mysql.url="jdbc:mysql://SERVER/DB?serverTimezone=Europe/Oslo"
db.mysql.username=USER
db.mysql.password=PASS
play.evolutions.db.default.autoApply=true
play.evolutions.db.mysql.autoApply=true
db.mysql.enabled=true
ebean.mysql=["models.*"]

# user behavior tracking
userTracking=false

`secret/pipeline-webui/prod.env`

DAISY_PIPELINE2_URL=http://localhost:8181/ws

secret/pipeline-engine/prod.env

(hostname removed, replace with your own hostname)

PIPELINE2_WS_HOST=PUBLICHOSTNAME
PIPELINE2_WS_LOCALFS=false
PIPELINE2_WS_AUTHENTICATION=false
PIPELINE2_WS_AUTHENTICATION_KEY=key
PIPELINE2_WS_AUTHENTICATION_SECRET=secret
JAVA_MAX_MEM=4G

martinpub commented 2 years ago

Thank you very much for this detailed description @josteinaj! For the documentation, I went with a simple docker-compose configuration. I have also noted the restart/resume issue and ways to work around it.

martinpub commented 2 years ago

Regarding the main topic of this issue, it can be considered partly solved with #511. In my mind, much of the user guide that @josteinaj has made is still applicable. Perhaps there is still a need for a straight-forward validation guide? I can fix this in the coming days.

Ping @AndersEkl

nlbdev / nordic-epub3-dtbook-migrator