matrix-registration pgloader fails to build on arm32 (Raspberry Pi 4)

[whowantsmybigdata]

manually I can build pgloader but only if I command make DYNSIZE=2048 pgloader. My Raspberry Pi has 4gb of memory so the default --dynamic-space-size=4096 is failing. probably it would be the best to add a variable which looks for the available memory and puts it in the build-config. sorry as I'm not able to do that on my own, I really looked for that part in the roles of the playbook but couldnt find anything...

[spantaleev]

pgloader building happens in roles/matrix-postgres/tasks/util/migrate_db_to_postgres.yml via a docker_image command: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/roles/matrix-postgres/tasks/util/migrate_db_to_postgres.yml#L60-L69

Actual building happens in a container using this Dockerfile.

We already patch the Dockerfile ourselves a little bit, here: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/f5a2d732ea08a78cfe1d4bb8c81ae8851e812ab5/roles/matrix-postgres/tasks/util/migrate_db_to_postgres.yml#L43-L58

We might add some similar workaround above/below this task, which replaces && make with && make DYNSIZE=2048 (or whatever the correct command is). The new Ansible task could be made to only run when: "matrix_architecture == 'arm32'".

It'd be best if this is fixed upstream, but seeing how we have other workarounds in place for very old issues, it's unlikely anybody would care.

[whowantsmybigdata]

ok I tried it but failed... in matrix-docker-ansible-deploy/roles/matrix-postgres/tasks/util/migrate_db_to_postgres.yml I insert:

    - name: Change dynamic-space-size to 2048 on arm32
      lineinfile:
        path: "{{ matrix_postgres_pgloader_container_image_self_build_src_path }}/Dockerfile"
        regexp: "{{ item.match }}"
        line: "{{ item.replace }}"
      with_items:
        - match: '^RUN git clone --branch ${PGLOADER_VERSION} --depth 1 https://github.com/dimitri/pgloader /opt/src/pgloade$
                 && mkdir -p /opt/src/pgloader/build/bin \
                 && cd /opt/src/pgloader \
                 && make$'
          replace: 'RUN git clone --branch ${PGLOADER_VERSION} --depth 1 https://github.com/dimitri/pgloader /opt/src/pgload$
                   && mkdir -p /opt/src/pgloader/build/bin \
                   && cd /opt/src/pgloader \
                   && make DYNSIZE=2048 pgloader'

Still there is no change:

FAILED! => {"changed": false, "msg": "Error building localhost/illagrenan/pgloader - code: 2, message: The command '/bin/sh -c git clone --branch ${PGLOADER_VERSION} --depth 1 https://github.com/dimitri/pgloader /opt/src/pgloader     && mkdir -p /opt/src/pgloader/build/bin     && cd /opt/src/pgloader     && make' returned a non-zero code

so its still doing a simple make without the correction of DYNSIZE

btw: I guess this issue is on every architecture in case you self-build and have less than 4096mb of memory, isn't it?

[spantaleev]

The above match and replace will never match. You're trying to do some complicated multi-line regular expression match, while lineinfile only supports individual lines.

Try match = && make$ and replace = && make DYNSIZE=2048 pgloader.

If you take a look at the /matrix/postgres/pgloader-container-src/Dockerfile file on the server, you'll know if it worked or not.

---

btw: I guess this issue is on every architecture in case you self-build and have less than 4096mb of memory, isn't it?

I suppose it has to do with the amount of addressable memory on that CPU architecture, and not with the amount that is actually installed.

[whowantsmybigdata]

Try match = && make$ and replace = && make DYNSIZE=2048 pgloader.

Yes. That worked!! Thanks a lot! Should I commit that with the when: "matrix_architecture == 'arm32'"?

I suppose it has to do with the amount of addressable memory on that CPU architecture, and not with the amount that is actually installed.

ok, I m absolutely too much of a noob to suppose anything...

[whowantsmybigdata]

ok that was too early: TASK [Import sqlite database from /matrix/matrix-registration/data/db.sqlite3 into Postgres] failed with:

FAILED! => {"changed": true, "cmd": ["/usr/bin/env", "docker", "run", "--rm", "--user=1001:1001", "--cap-drop=ALL", 
"--network=matrix", "--mount", "type=bind,src=/matrix/matrix-registration/data/db.sqlite3,dst=/in.db,ro", 
"--entrypoint=/bin/sh", "localhost/illagrenan/pgloader:3.6.2", "-c", 
"pgloader  /in.db postgresql://matrix_registration:4cfe0402-e0b0-568f-ba25-80fbfdc2e6bb@matrix-postgres:5432/matrix_registration"], 
"delta": "0:00:03.211106", "end": "2021-08-16 18:37:10.910825", "msg": "non-zero return code", "rc": 1, "start": 
"2021-08-16 18:37:07.699719", "stderr": "KABOOM!\nFATAL error: There is no thread support in this instance.\n
An unhandled error condition has been signalled:\n   There is no thread support in this instance.\n\n\n\n\nWhat I am doing
here?\n\nThere is no thread support in this instance.", "stderr_lines": ["KABOOM!", "FATAL error: There is no thread support in
this instance.", "An unhandled error condition has been signalled:", "   
There is no thread support in this instance.", "", "", "", "", "What I am doing here?", "", 
"There is no thread support in this instance."], "stdout": "", "stdout_lines": []}