MI-DPLA / combine-docker

Combine Docker-ized
https://mi-dpla.github.io/combine-docker/
6 stars 12 forks source link

handle versioning of Combine as a system in Combine-Docker #17

Closed ghukill closed 5 years ago

ghukill commented 5 years ago

Closed https://github.com/WSULib/combine-docker/issues/16 without much fanfare, as it really only scratched the surface of a larger question of how to handle versions and upgrades of Combine.

The Combine and Combine-Playbook repositories are both versioned -- via tagged releases -- that conspire to create a more abstract version to Combine as a system generally. This repository can do the same, and arguably even more effectively, by also having tagged releases.

This dockerized version jumps in on the heels of v0.5.2, and seems like a reasonable v0.6 release of Combine. This will mean the first release of this repository is also v0.6.

A tagged release will have distinct code in areas like:

The biggest challenge will be volumes that endure re-builds of containers and images, without losing data in Combine. While exporting an re-importing is an option, it's not ideal.

Of the following volumes defined in docker-compose.yml:

volumes:
  esdata:
    driver: local
  mongodata:
    driver: local
  mysqldata:
    driver: local
  hdfs:
    driver: local
  combine_home:
    driver: local
  combine_python_env:
    driver: local
  combine_django_app:
    driver: local
  hadoop_binaries:
    driver: local
  spark_binaries:
    driver: local
  livy_binaries:
    driver: local
  combine_tmp:
    driver: local

It's necessary to remove volumes such as combine_django_app, as even though re-builds of the combine-django image and containers might pull new and different branches/tags, the old volume would re-mount and project the old code over the newly constructed image.

Only the following would be required to save to maintain state between upgrades:

esdata
mongodata
mysqldata
combine_home

So, then, is it unreasonable to expect users to purge some volumes, but not all, when upgrading?

ghukill commented 5 years ago

This has been addressed by: