webrecorder / browsertrix

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!
https://webrecorder.net/browsertrix
GNU Affero General Public License v3.0
201 stars 35 forks source link

[Bug]: Error creating WACZ - ETIMEDOUT #2121

Open tkrn opened 3 weeks ago

tkrn commented 3 weeks ago

Browsertrix Version

v1.12.0-6032e28

What did you expect to happen? What happened instead?

When I have an external defined minio instance and when the crawl job moves to the upload/creating warz phase, it fails with the following error: "Error creating WACZ" with a more verbose error of "Error: connect ETIMEDOUT 10.215.7.8:9000". With that being said, the credentials are correct and I am able to enumerate the S3 bucket through the minio client (mc) with success on the same box with the same host and access/secret key.

Reproduction instructions

  1. Create a custom S3 storage target as follows:
    minio_local: false
    storages:
    - name: "default"
    type: "s3"
    region: "docker"
    access_key: "<access>"
    secret_key: "<secret>"
    bucket_name: "btrix"
    endpoint_url: "http://<ip>:9000/"
  2. Create a new crawl job. Wait for error.

Screenshots / Video

Error Message: thunderbolt-btrix-error

local-config.yaml for Browertrix: thunderbolt-local-config-yaml

Minio Client Connection Parameters: thunderbolt-mc-config

Minio Client Successful Connection: thunderbolt-mc-ls

Environment

No response

Additional details

Both the MongoDB instance and Minio instance are running on the same physical server but through docker compose. The ports are listening to 0.0.0.0:9000 allowing any connections including hairpin and localhost connections. This configuration is working successfully for the MongoDB instance which is outside of the Browsertrix deployment, although physically on the same server but logically outside of the microk8s cluster. The be stated, the MongoDB instance is working as expected with this type of deployment.