erikjohnston
commented
1 month ago https://github.com/element-hq/synapse/pull/17228 was a quick attempt to address this. The approach was to basically add a mounted directory to the python path, and then doing pip install -t <dir> <package> to install packages into it.
This worked fine for pure python packages, but breaks down in the presence of native packages. This is due to the native packages being built for the host machine rather than the docker image, which could cause all sorts of woe (especially if the python version of the docker image changes after an upgrade). This is particularly bad because it may work some of the time, and then randomly break.
We could just require people to create new docker images if they want to, and specify a directory to install into during build. This should be a bit more robust than what some people do today where they install into the same virtualenv as Synapse.
Otherwise, we could add a install-package <package> command to the docker image (or something similar) that will build/install the package correctly. These would be installed to a directory on a mounted value so persist across restarts. We would probably want to make that directory include the python version (which is what all the standard python paths have), so that upgrading python doesn't silently break things. This would mean that the packages would need to be reinstalled when upgrade the docker images.
This issue has been migrated from #9944.
In the process of splitting the Account Validity feature into its own module we've run up against a roadblock: we don't have a well-defined path for people to add extension modules to our Docker images.
We need to survey prior art, pick a solution, and then implement and document it.