search

matrix-org / synapse-s3-storage-provider

Synapse storage provider to fetch and store media in Amazon S3
Apache License 2.0
118 stars 33 forks source link

Steps to install #14

Open tompinzler opened 5 years ago

tompinzler commented 5 years ago

Sorry if this is a stupid question!

I'm on Debian 9 and did pip install git+https://github.com/matrix-org/synapse-s3-storage-provider.git

I add the following configuration to /etc/matrix-synapse/homeserver.yaml

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
   bucket: matrix

Then I did a systemctl restart matrix-synapse

In syslog I see

matrix python[6899]: ImportError: No module named s3_storage_provider

Tow questions:

  1. How do I have to install synapse-s3-storage-provider so Matrix can find/use it?

  2. How/Where do I configure my S3 credentials?

abeluck commented 5 years ago

Also looking for an answer to this!

Sharparam commented 5 years ago

Late to the party but could be useful to others finding this issue: It seems it will pick up credentials that have previously been saved with the AWS CLI tool.

I installed it in the same virtual env I use for synapse and authenticated with the tokens for the synapse-specific IAM I made in the AWS console.

For system-installed Synapse I'm not sure what the process would be.

Edit: And also the steps outlined in the boto3 docs which I realised are actually linked in the readme of this project.

Edit 2: Also for Synapse to find the tool I just installed this project via pip while I had the virtual env for synapse active.

bradinator commented 5 years ago

Steps to install would be great. I'm having trouble getting it all working.

rkfg commented 5 years ago

I think you've installed the storage provider for Python 2 while Synapse uses Python 3. I bumped into the same issue. Install pip for Python 3: apt install python3-pip then do pip3 install git+https://github.com/matrix-org/synapse-s3-storage-provider.git and proceed with the README instructions. I did that and it works fine it seems, I can see the new files in the bucket.

bradinator commented 5 years ago

@rkfg thanks for that. I'm probably a step closer now. However, I'm still running into an issue anytime I update the config to include the storage provider and restart synapse, it just continually crashes until I remove those lines again.

rkfg commented 5 years ago

You should check the logs then (like /var/log/daemon.log), maybe you miss a module.

bradinator commented 5 years ago

I checked for logs, @rkfg i don't find a daemon.log but this is from my homeserver.log This seems to be the issues, but i don't know what to make of it.

`2019-07-02 11:13:25,582 - twisted - 242 - INFO - - Received SIGTERM, shutting down. 2019-07-02 11:13:25,582 - synapse.handlers.presence - 244 - INFO - presence.on_shutdown-0- Performing _on_shutdown. Persisting 25 unpersisted changes 2019-07-02 11:13:25,590 - synapse.handlers.presence - 253 - INFO - presence.on_shutdown-0- Finished _on_shutdown 2019-07-02 11:13:25,591 - synapse.http.site - 203 - WARNING - GET-8991- Error processing request <XForwardedForRequest at 0x7fc75566b8d0 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722238_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost. 2019-07-02 11:13:25,592 - synapse.http.site - 203 - WARNING - GET-8977- Error processing request <XForwardedForRequest at 0x7fc748443cc0 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722238_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.

2019-07-02 11:13:25,597 - twisted - 242 - INFO - - (TCP Port 8008 Closed) 2019-07-02 11:13:25,597 - synapse.http.site - 203 - WARNING - GET-9003- Error processing request <XForwardedForRequest at 0x7fc729d3b6d8 method='GET' uri='/_matrix/client/r0/sync?filter=1&timeout=30000&since=s168684_2722261_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost. 2019-07-02 11:13:25,598 - synapse.http.site - 203 - WARNING - GET-8979- Error processing request <XForwardedForRequest at 0x7fc729b02400 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722238_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.

2019-07-02 11:13:25,600 - twisted - 242 - INFO - - (TCP Port 8008 Closed) 2019-07-02 11:13:25,600 - synapse.http.site - 203 - WARNING - GET-9011- Error processing request <XForwardedForRequest at 0x7fc729a53668 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722261_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.

2019-07-02 11:13:25,606 - twisted - 242 - INFO - - (TCP Port 8448 Closed) 2019-07-02 11:13:25,607 - synapse.http.site - 203 - WARNING - GET-8969- Error processing request <XForwardedForRequest at 0x7fc74811dbe0 method='GET' uri='/_matrix/client/r0/sync?filter=0&timeout=30000&since=s168684_2722238_80_232366_330477_125_88026_1208_2' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost. 2019-07-02 11:13:25,617 - twisted - 242 - INFO - - Main loop terminated.`

rkfg commented 5 years ago

Does your Synapse work fine without this S3 backend? Post the full log, this part only says it's got SIGTERM and terminates but it's not clear why.

bradinator commented 5 years ago

Yes, synapse works fine without the s3 backend. As soon as I change out the storage provider lines in the homeserver.yaml and restart, clients can't connect, the status indicates that it continues to attempt restarting.

The full log is 300k lines, so I was trying to only post from the timestamp I restarted, to when I restarted with no s3 backend. How much of the log should i provide?

rkfg commented 5 years ago

Just zip it up and attach here. Or post more lines because the first line in your snippet says that Synapse received SIGTERM (which is an external event like stopping the systemd unit) and the rest looks like cleaning up from that. I don't see anything related to its startup.

rkfg commented 5 years ago

I don't see anything unusual there. Can you produce a clean startup log by removing whatever log files you already have and starting Synapse with S3 enabled?

bradinator commented 5 years ago

Steps I tried to grab a better log:

  1. Replace homeser.yaml with one pointing to S3
  2. Clear Log
  3. Restart Matrix
  4. See that it isn't starting correctly,
  5. Download logs provided below.

homeserver2.zip

rkfg commented 5 years ago

Failed to insert client IP

This is a weird error, I don't have it. Does it happen when Synapse works without S3? Is the only difference between those configs you swap S3 being enabled and not? Also, post that config (without passwords and such of course). Maybe the problem is there.

bradinator commented 5 years ago

Yes, I get the Failed to insert client IP while synapse is working properly without S3.

Everything else is identical. 

# Directory where uploaded images and attachments are stored.
##
#media_store_path: "/var/lib/matrix-synapse/media"

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: riot

# Directory where in-progress uploads are stored.
#
uploads_path: "/var/lib/matrix-synapse/uploads"
rkfg commented 5 years ago

Here's mine:

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_synchronous: False
  config:
    bucket: "synapse-bc"
    access_key_id: "key_id"
    secret_access_key: "access_key"
    storage_class: "INTELLIGENT_TIERING"

I suppose you have access_key_id and secret_access_key parameters set. How do you run Synapse? Do you use Systemd/SysV or anything like that (if yes, check the system logs not just Synapse's own logs; some errors are not visible there)? Have you tried to run it manually (using the same command the system uses + sudo to run as matrix-synapse user)?

bradinator commented 5 years ago

Does this mean anything useful? As the matrix-synapse user, I went in to python3 help, and found the module "s3_storage_provider". But when I give the command for help on that module, i see the below message. 

help> s3_storage_provider
problem in s3_storage_provider - ModuleNotFoundError: No module named 'synapse'
rkfg commented 5 years ago

Could be a path problem then. Where's your Synapse installed to (path)? I use the official apt repository, it was a bit tricky to install this S3 module because they provide their own python and libraries with Synapse.

bradinator commented 5 years ago

I used this guide when installing, so I used apt to install. https://www.natrius.eu/dokuwiki/doku.php?id=digital:server:matrixsynapse

rkfg commented 5 years ago

Are you using matrix-synapse-py3 or matrix-synapse? The former comes with its own Python and libraries so you should use pip from there to install the S3 provider properly.

HelderFSFerreira commented 4 years ago

What about the install process in docker? As the package is not installed on the official synapse docker image, there is an alternative or a PR to the official docker image should be made?

ShadowJonathan commented 4 years ago

What about the install process in docker?

I also really wanna know this, as I handle a personal synapse server right now, and after adding a whatsapp bridge, the media store started bloating significantly, i'd want to move that to b2, and do the same with other synapse servers in the future

ShadowJonathan commented 4 years ago

After I looked at the readme real quick, i noticed the only "installation" step:

The s3_storage_provider.py should be on the PYTHONPATH when starting synapse.

This means that doing something like mounting or placing it in the "/data" folder (if that's the working directory of synapse while running) would be enough to "install" it, though i think i'd be ignoring circumstances and dependencies at that point

iokiwi commented 3 years ago

@ShadowJonathan @CaptainYukinoshitaHachiman I had this question too. I think the best option is to build and image based on the official matrix/synapse image and install the synapse-s3-storage-provider during the image build. Doing a pip install . inside the repository should build, install and add synapse-s3-storage-provider to the expected path(s).

It should be almost as simple as either cloning or copying the repository into the image, entering the folder and doing a pip install .

Eg. Cloning

FROM matrixdotorg/synapse:latest

RUN git clone https://github.com/matrix-org/synapse-s3-storage-provider.git && \
        cd /synapse-s3-storage-provider && \
        pip install .

E.g COPY

FROM matrixdotorg/synapse:latest

COPY synapse-s3-storage-provider /synapse-s3-storage-provider

RUN cd /synapse-s3-storage-provider && \
        pip install .

In practice you may need to also install some missing libraries / dependencies in order for pip to be able to install synapse-s3-storage-provider..

Here's a Dockerfile i've been using to test an Openstack Swift Storage provider I am trying to build based off the AWS S3 storage provider.

FROM matrixdotorg/synapse:latest

# COPY synapse-swift-storage-provider /synapse-swift-storage-provider

RUN apk update && \
    apk add --no-cache \
      g++ \
      libffi-dev \
      git \
      python3-dev && \
    pip install --upgrade pip && \
    git clone https://github.com/iokiwi/synapse-swift-storage-provider.git && \
    cd /synapse-swift-storage-provider && \
    pip install . && \
    mkdir -p  /root/.config/openstack && \
    apk del \
        git \
        g++ \
        libffi-dev \
        python3-dev && \
    rm -rf /var/cache/apk/* && \
    rm -rf /tmp/*

# COPY clouds.yaml /root/.config/openstack/

Note I had to install a lot of extras (mostly for the openstacksdk). Unsure about dependencies for boto3 and botocore sorry.

Further installing extra depencies, I'd also recommend uninstalling as many of the dependencies as you can at the end of the build process to reduce the weight of the image and reduce the number of unnecessary binaries included.

jimmybrancaccio commented 3 years ago

I was able to get this working by building my own image.

FROM matrixdotorg/synapse:latest

RUN apt -y update \
    && apt -y upgrade \
    && apt -y install g++ libffi-dev git python3-dev python3-pip \
    && pip install --upgrade pip \
    && git clone https://github.com/iokiwi/synapse-swift-storage-provider.git \
    && cd /synapse-swift-storage-provider \
    && pip install . \
    && pip3 install git+https://github.com/matrix-org/synapse-s3-storage-provider.git

and my homeserver.yml:

media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: my.bucketname.com
    region_name: us-east-1
    access_key_id: key-id-here
    secret_access_key: key-here

You may need to adjust the region_name depending on the region of your S3 bucket.

wioxjk commented 3 years ago

Why is this module not included in the .deb package yet?

inaniyants commented 3 years ago

Hi! In case you have missed, this package was published to pip registry on May 18, 2021 https://pypi.org/project/synapse-s3-storage-provider/ But this package publication is not automated (not in sync with this repo), so use next on your own risk.

So now you could install it by simply running pip install synapse-s3-storage-provider.

This means, that starting from now you don't need to install git via apk, or apt Here is working Dockerfile

FROM matrixdotorg/synapse:v1.34.0

RUN pip install synapse-s3-storage-provider
Ibrahim-AbouElseoud commented 3 years ago

For those using Yunohost to install this module, I would like to add the following steps which had worked for me:

  1. SSH to your yunohost server
  2. Login as root or dosudo -i if logged in with admin
  3. Activate the python venv of the synapse app which is located in /opt/yunohost/matrix-synapse/ by the following command: source /opt/yunohost/matrix-synapse/bin/activate
  4. Install the module with pip: pip install synapse-s3-storage-provider
  5. Add the required settings in your homeserver.yaml which is located in /etc/matrix-synapse/homeserver.yaml, the new settings to insert should look like the following:
media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: "your_bucket_name"
    # All of the below options are optional, for use with non-AWS S3-like
    # services, or to specify access tokens here instead of some external method.
    region_name: "your-region-name"
    endpoint_url: "https://your-storage.com"
    access_key_id: "your_key"
    secret_access_key: "your_key"
Kitanit commented 2 years ago

@Ibrahim-AbouElseoud

For those using Yunohost to install this module, I would like to add the following steps which had worked for me:

  1. SSH to your yunohost server
  2. Login as root or dosudo -i if logged in with admin
  3. Activate the python venv of the synapse app which is located in /opt/yunohost/matrix-synapse/ by the following command: source /opt/yunohost/matrix-synapse/bin/activate
  4. Install the module with pip: pip install synapse-s3-storage-provider
  5. Add the required settings in your homeserver.yaml which is located in /etc/matrix-synapse/homeserver.yaml, the new settings to insert should look like the following:
media_storage_providers:
- module: s3_storage_provider.S3StorageProviderBackend
  store_local: True
  store_remote: True
  store_synchronous: True
  config:
    bucket: "your_bucket_name"
    # All of the below options are optional, for use with non-AWS S3-like
    # services, or to specify access tokens here instead of some external method.
    region_name: "your-region-name"
    endpoint_url: "https://your-storage.com"
    access_key_id: "your_key"
    secret_access_key: "your_key"

Sorry for the question, after uploading files to the S3 repository which url will handle the file, synapse instance or repository, just configured everything from the posts above, but confused that everything is uploaded to S3, but I do not notice the reverse action *As an exact solution, is there a clause in the config to proxy everything through the S3 domain?

sorcer1122 commented 1 year ago

I am still getting No module named s3_storage_provider on Debian 11. I am using matrix-synapse-py3 and did pip3 install synapse-s3-storage-provider.

Any suggestions where to dig?

Ok for those who face the same issue - I copied all the necessary folders from /usr/local/lib/python3.9 to /opt/venvs/matrix-synapse/lib/python3.9 manually and all worked!

gityzon commented 1 year ago

If you use command apt install matrix-synapse-py3 to install synapse then try /opt/venvs/matrix-synapse/bin/pip3 install synapse-s3-storage-provider to install synapse-s3-storage-provider. Hope this may help someone.

ifedan-ed commented 1 year ago

If you use command apt install matrix-synapse-py3 to install synapse then try /opt/venvs/matrix-synapse/bin/pip3 install synapse-s3-storage-provider to install synapse-s3-storage-provider. Hope this may help someone.

This works for the DO app. Thanks!