Docker containers for arm platforms

shoopdawoop commented 4 months ago

I was wondering if it is possible to build containers for sal and postgres-sal for arm64 platforms in order to host the whole thing on modern/efficient hardware. I would build the containers myself if I know how to do it.

Ideally, ready made containers for arm64 would be provided in addition to the amd64 containers, but of course I'd be willing to do some work myself.

grahamgilbert commented 4 months ago

Sal hasn't been tested at all on Arm. If you want to do a manual installation first and verify it works that is the first step. Also, you should not use the postgres container for anything more than a demo. It has not been updated for years.

Note: For a production setup, it is recommended that you do not use macOS as your host OS or run Postgres in Docker

shoopdawoop commented 4 months ago

Thank you.

So I set up a Linux VM on an M2 Mac in VMWare Fusion to test this and Sal seems to run just fine.

I was able to enrol an older macOS 12 Client VM and collect some data and everything seemed to work as expected.

The test server setup (I followed the instructions for Ubuntu 16):

Ubuntu Server 22.04.4 LTS (GNU/Linux 5.15.0-113-generic aarch64) Python 3.10.12 git checkout tags/4.2.0

In only ran into minor issues while installing the python packages via requirements.txt:

The PyYAML and gevent packages were throwing an error (see below) but I found a workaround here https://stackoverflow.com/questions/77490435/attributeerror-cython-sources (via https://github.com/yaml/pyyaml/issues/601):

After installing these packages first

pip install "cython<3.0.0" wheel
pip install "pyyaml==5.4.0" --no-build-isolation
pip install "gevent==21.1.2" --no-build-isolation

another

pip install -r requirements.txt

finished without errors.

Also, you should not use the postgres container for anything more than a demo. It has not been updated for years.

I get that also running the postgres-sal container in production is less than ideal, but I have been running both Sal and the old postgres-sal as containers for quite a while now without any issues. And for reasons™ *) beyond running Sal in Docker being pretty convenient, obviously, if possible, I would like to continue doing so.

(* two of the reasons being running a relatively small scale setup and running Sal on a shared host)

So, how would I go from here?

Example pip error:

Collecting PyYAML==5.4 (from -r setup/requirements.txt (line 14))
  Downloading PyYAML-5.4.tar.gz (174 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174.8/174.8 kB 4.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/usr/local/sal_install/sal_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/sal_install/sal_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/usr/local/sal_install/sal_env/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 974, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-3dhwkd9z/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

grahamgilbert commented 4 months ago

As I have no need to run this on Arm, I am unable to commit any time to getting this working. If you can get it all working and commit to supporting it in the future, I will accept a pull request for it. My container infrastructure is still x86 so until that changes, I cannot support this. I will leave this issue open with the help wanted tag on.

shoopdawoop commented 3 months ago

I dont feel confident I could/should maintain this (I was not even confident I would get this far thb), but I can report that I have successfully managed to

create arm64 container images for sal and postgres-sal with only small changes to the existing Dockerfiles (see below)
run the arm64 images in docker on an arm64 Ubuntu 22.04.4 Server
setup up a new Sal instance from scratch, create a Business Unit and have a macOS 12x Client VM report to Sal

I was not yet able to dump an existing db from the old 9.6 container, upgrade the db and restore it to the 15.2 container. I might look into this later ...

So in case anyone else might want to keep going from here:

Dockerfile-postgres-arm64.txt Dockerfile-sal-arm64.txt

Dockerfile-postgres-arm64 is simply based on postgres:15.2,

Dockerfile-sal-arm64 now includes

RUN pip install --upgrade pip
RUN pip install "cython<3.0.0" wheel
RUN pip install "pyyaml==5.4.0" --no-build-isolation
RUN pip install "gevent==21.1.2" --no-build-isolation

salopensource / sal

Docker containers for arm platforms #461