S3 compatible service url mismatch

[cerri]

Hello, using the S3 plugin and selecting a compatible server, the url tried is wrong. All settings are fine, so I can assume this is a bug.

Thanks

[jcameron]

Does that "could not connect to the endpoint" error appear on the cloud backup destination page, or when you try to make a backup?

[cerri]

During the backup. That error appears in the backup log.

[jcameron]

Which Virtualmin version are you running there?

[cerri]

Sorry, I forgot - 7.7-3

[jcameron]

Do you have the aws command installed on your system?

If so, can you use it to list buckets on this service by running something like aws --endpoint-url=https://backupserver3.init-s.it s3 ls ?

[cerri]

Yes, currently working on command line.

root@hl01:~# aws --endpoint-url=https://backupservers3.init-s.it s3 ls
2022-06-14 08:31:37 hl01.clienti.init-s.it
root@hl01:~#

[jcameron]

Ok that seems fine..

Can you post the full backup log so I can see where in the process it's failing?

Also, what output do you get if you SSH in and run virtualmin list-s3-buckets ?

[cerri]

Hello,

here the logs when it was running fine (until 04/26) and when it started to fail.

04/27/2023

Testing backup destination 27-04-2023 in Amazon S3 bucket hl01.clienti.init-s.it ..
.. done

Creating backup for virtual server highlinestore.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database highlinestore ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Backing up AWStats configuration file ..
    .. done

    Backing up DAV users file ..
    .. done

    Creating TAR file of home directory ..
    .. done

    Uploading archive to Amazon's S3 service ..
    .. upload failed! upload failed: ../../../../tmp/.webmin/27313-/highlinestore.com.tar.gz to s3://hl01.clienti.init-s.it/27-04-2023/highlinestore.com.tar.gz Could not connect to the endpoint URL: "https://s3.inits-de-hetzner-1.amazonaws.com/hl01.clienti.init-s.it/27-04-2023/highlinestore.com.tar.gz?uploads"

.. completed in 3 minutes, 9 seconds

Creating backup for virtual server highlinestore.it ..
    Copying virtual server configuration ..
    .. done

    Copying Apache aliases ..
    .. done

    Creating TAR file of home directory ..
    .. done

    Uploading archive to Amazon's S3 service ..
    .. upload failed! upload failed: ../../../../tmp/.webmin/27313-/highlinestore.it.tar.gz to s3://hl01.clienti.init-s.it/27-04-2023/highlinestore.it.tar.gz Could not connect to the endpoint URL: "https://s3.inits-de-hetzner-1.amazonaws.com/hl01.clienti.init-s.it/27-04-2023/highlinestore.it.tar.gz"

.. completed in 47 seconds

Creating backup for virtual server hl01.clienti.init-s.it ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Creating TAR file of home directory ..
    .. done

    Uploading archive to Amazon's S3 service ..
    .. upload failed! upload failed: ../../../../tmp/.webmin/27313-/hl01.clienti.init-s.it.tar.gz to s3://hl01.clienti.init-s.it/27-04-2023/hl01.clienti.init-s.it.tar.gz Could not connect to the endpoint URL: "https://s3.inits-de-hetzner-1.amazonaws.com/hl01.clienti.init-s.it/27-04-2023/hl01.clienti.init-s.it.tar.gz"

.. completed in 55 seconds

04/26/2023

Creating backup for virtual server highlinestore.com ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Webalizer configuration files ..
    .. done

    Copying SSL Apache virtual host configuration and certificate ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Dumping MySQL database highlinestore ..
    .. done

    Backing up Webmin ACL files ..
    .. done

    Backing up AWStats configuration file ..
    .. done

    Backing up DAV users file ..
    .. done

    Creating TAR file of home directory ..
    .. done

    Uploading archive to Amazon's S3 service ..
    .. done

.. completed in 3 minutes, 10 seconds

Creating backup for virtual server highlinestore.it ..
    Copying virtual server configuration ..
    .. done

    Copying Apache aliases ..
    .. done

    Creating TAR file of home directory ..
    .. done

    Uploading archive to Amazon's S3 service ..
    .. done

.. completed in 3 seconds

Creating backup for virtual server hl01.clienti.init-s.it ..
    Copying virtual server configuration ..
    .. done

    Backing up Cron jobs ..
    .. none defined.

    Saving mail aliases ..
    .. done

    Saving mail and FTP users ..
    .. done

    Backing up mail and FTP user Cron jobs ..
    .. none to backup

    Copying Apache virtual host configuration ..
    .. done

    Copying Apache log files ..
    .. done

    Copying Logrotate configuration ..
    .. done

    Creating TAR file of home directory ..
    .. done

    Uploading archive to Amazon's S3 service ..
    .. done

.. completed in 4 seconds

3 servers backed up successfully, 0 had errors.

Deleting backups from local file /usr/share/webmin/virtual-server/hl01.clienti.init-s.it matching .*-.*-.* older than 30 days ..
    Deleting file 26-03-2023/highlinestore.com.tar.gz, which is 31 days old ..
    .. deleted 1.52 GiB.

    Deleting file 26-03-2023/highlinestore.it.tar.gz, which is 31 days old ..
    .. deleted 1.25 KiB.

    Deleting file 26-03-2023/hl01.clienti.init-s.it.tar.gz, which is 31 days old ..
    .. deleted 178.45 KiB.

.. deleted 3 old backups, and skipped 92 that were not old enough

Here data you requested in ssh.

root@hl01:~# virtualmin list-s3-buckets
Bucket name                                   Created
--------------------------------------------- ------------------------------
hl01.clienti.init-s.it                        06/14/2022 08:31 AM
root@hl01:~#

Thank you

[jcameron]

It seems like the issue is that the aws command isn't working properly with your alternate S3-compatible service.

What happens if you run a command like this to manually upload a file :

aws --endpoint-url=https://backupservers3.init-s.it s3 cp /tmp/highlinestore.it.tar.gz s3://hl01.clienti.init-s.it/27-04-2023/highlinestore.it.tar.gz

[cerri]

Hello,

it's working fine (I've just uploaded a test file).

root@hl01:/tmp# aws --endpoint-url=https://backupservers3.init-s.it s3 cp /tmp/settings.inc.php s3://hl01.clienti.init-s.it/27-04-2023/
upload: ./settings.inc.php to s3://hl01.clienti.init-s.it/27-04-2023/settings.inc.php
root@hl01:/tmp#

This is the command virtualmin is running: /usr/bin/python3 /usr/bin/aws s3 --profile=hl01 --endpoint-url=https://backupservers3.init-s.it cp /tmp/.webmin/20497-/hl01.clienti.init-s.it.tar.gz s3://hl01.clienti.init-s.it/16-05-2023/hl01.clienti.init-s.it.tar.gz

So I tried to do the same command as before but with python interpreter:

root@hl01:~/.aws# /usr/bin/python3 /usr/bin/aws s3 --profile=hl01 --endpoint-url=https://backupservers3.init-s.it cp /tmp/settings.inc.php s3://hl01.clienti.init-s.it/27-04-2023/
upload failed: ../../../tmp/settings.inc.php to s3://hl01.clienti.init-s.it/27-04-2023/settings.inc.php Could not connect to the endpoint URL: "https://s3.inits-de-hetzner-1.amazonaws.com/hl01.clienti.init-s.it/27-04-2023/settings.inc.php"
root@hl01:~/.aws#

and here the error. I hope it helps.

[jcameron]

Is it the /usr/bin/python3 interpreter that causes the issue, or the --profile=hl01 flag?

[cerri]

It's the profile flag.

root@hl01:~# /usr/bin/python3 /usr/bin/aws s3 --endpoint-url=https://backupservers3.init-s.it cp /tmp/settings.inc.php s3://hl01.clienti.init-s.it/27-04-2023/
upload: ../../tmp/settings.inc.php to s3://hl01.clienti.init-s.it/27-04-2023/settings.inc.php
root@hl01:~#

[jcameron]

In /etc/webmin/virtual-server/config , do you have the aws_cmd set to include that --profile=hl01 flag?

[cerri]

No.

root@hl01:~# cat /etc/webmin/virtual-server/config  | grep aws_cmd
aws_cmd=aws

Please note that I have several Ubuntu+Virtualmin instances with the same problem and I'm using just one of them to investigate the problem.

[jcameron]

What does the hl01 section of /root/.aws/credentials contain?

[cerri]

Just access_key_id and secret_access_key

[hl01]
aws_access_key_id = xxxxxx
aws_secret_access_key = xxxxxx

[jcameron]

What about the default profile - is it using the same keys?

[cerri]

Hello,

yes.

root@hl01:~# aws configure
AWS Access Key ID [****************hl01]: hl01
AWS Secret Access Key [****************v1PH]: ****************v1PH
Default region name [inits-de-hetzner-1]:
Default output format [None]:
root@hl01:~#

root@hl01:~# aws configure --profile=hl01
AWS Access Key ID [****************hl01]: hl01
AWS Secret Access Key [****************v1PH]: ****************v1PH
Default region name [None]: inits-de-hetzner-1
Default output format [None]:
root@hl01:~#

Anyway, after setting again it on command line (as shown above) the command succeded. Running the backup from Virtualmin, it failed again.

Creating backup for virtual server highlinestore.it ..
Copying virtual server configuration ..
.. done
Copying Apache aliases ..
.. done
Creating TAR file of home directory ..
.. done
Uploading archive to Amazon's S3 service ..
.. upload failed! upload failed: ../../../../tmp/.webmin/6612-/highlinestore.it.tar.gz to s3://hl01.clienti.init-s.it/22-05-2023/highlinestore.it.tar.gz Could not connect to the endpoint URL: "https://s3.inits-de-hetzner-1.amazonaws.com/hl01.clienti.init-s.it/22-05-2023/highlinestore.it.tar.gz" 
.. completed in 53 seconds

[jcameron]

I wonder if that default region name of inits-de-hetzner-1 would be the issue?

[cerri]

Added the region in config file solved the problem.

root@hl01:/root# cat /root/.aws/config
[profile *************WR]
[profile hl01]
region=inits-de-hetzner-1
root@hl01:/root#

I think the region part must be revised (is this one?):

[jcameron]

Yeah, that region field makes no sense when not using Amazon S3 - we'll fix that.

As a work-around, try changing it to "Default".

[cerri]

It's "default" in all failing installation - it was just changed to take the shot, sorry about that.

[jcameron]

Ok, so it sounds like what we really need is a way to enter inits-de-hetzner-1 in that field as the default region..

[cerri]

Hello, any news? The backup is still failing on my side due this problem.

[jcameron]

Have you upgraded to the next Virtualmin? It should allow you to enter any string as the default region on the Cloud Backup Providers page.