[bug]: S3 backups broken

dolesec commented 2 weeks ago

FreePBX Version

FreePBX 17

Issue Description

looks to have been requested and reported already in issues 404 and 405

edge upgrade to filestore alleviated the issue of creating the store itself however the backup job still fails ...

Undefined array key "storageclass" Finished Remote Maintenance Saving to selected Filestore locations Undefined array key "storageclass" Finished Saving to selected Filestore locations There were errors during the backup process

Operating Environment

Asterisk 21.4.1

Relevant log output

...
Finished Cleaning up
Finished created backup file: 20240902-182821-1725316101-17.0.19.8-250340844.tar.gz
Performing Local Maintenance
Removing /var/spool/asterisk/backup/srqeyecare/20240902-181317-1725315197-17.0.19.8-1383201815.tar.gz
Finished Local Maintenance
Performing Remote Maintenance
Undefined array key "storageclass"
Finished Remote Maintenance
Saving to selected Filestore locations
Undefined array key "storageclass"
Finished Saving to selected Filestore locations
There were errors during the backup process
    Undefined array key "storageclass"
    Undefined array key "storageclass"
There were warnings during the backup process
    The module callback returned no data, No backup created
    The module callerid returned no data, No backup created
    The module calllimit returned no data, No backup created
    The module conferencespro returned no data, No backup created
    The module miscdests returned no data, No backup created
    The module setcid returned no data, No backup created
Generated Backup process result email to support@cditc.me. Status: Failure

dolesec commented 2 weeks ago

attempted to revert to filestore 17.0.2.22 and that failed as well

kguptasangoma commented 2 weeks ago

Hi @dolesec please try with filestore v17.0.2.27

jissphilip commented 2 weeks ago

Hi @dolesec Please try with this version of filestore 17.0.2.28

dolesec commented 2 weeks ago

different error and still broken - the same config was previously running successfully on my initial 17 test system and i noticed this when setting up a new production 17 system; i checked the my initial 17 test system and its broken there as well ...

new error reports the following:

Saving to selected Filestore locations
Unable to write file at location: mys3bucket/. Error executing "PutObject" on "https://s3pbxbackups.s3.amazonaws.com//mys3bucket/"; AWS HTTP error: Client error: `PUT https://s3pbxbackups.s3.amazonaws.com//mys3bucket/` resulted in a `400 Bad Request` response:

AccessControlListNotSupportedThe bucket does not all (truncated...)
 AccessControlListNotSupported (client): The bucket does not allow ACLs - 
AccessControlListNotSupportedThe bucket does not allow ACLs8778DJ2510KMRBRYKUC+OcVHl3pmHHeA5FRV5vqUtoAI4Io2d9GmVBjXtKQubHK2jV4Bw68vf+g6o4RWAXspEGouo0C88zsNfXtcBH1h1N4AFc+Hh78zCWySLaQ=
Finished Saving to selected Filestore locations
There were errors during the backup process
    Unable to write file at location: mys3bucket/. Error executing "PutObject" on "https://s3pbxbackups.s3.amazonaws.com//mys3bucket/"; AWS HTTP error: Client error: `PUT https://s3pbxbackups.s3.amazonaws.com//mys3bucket/` resulted in a `400 Bad Request` response:

AccessControlListNotSupportedThe bucket does not all (truncated...)
 AccessControlListNotSupported (client): The bucket does not allow ACLs - 
AccessControlListNotSupportedThe bucket does not allow ACLs8778DJ2510KMRBRYKUC+OcVHl3pmHHeA5FRV5vqUtoAI4Io2d9GmVBjXtKQubHK2jV4Bw68vf+g6o4RWAXspEGouo0C88zsNfXtcBH1h1N4AFc+Hh78zCWySLaQ=

dolesec commented 2 weeks ago

@kguptasangoma I see this was marked as closed, can it be reopened until we confirm the fix?

kguptasangoma commented 2 weeks ago

Hi @dolesec if you run v17.0.2.22 then does it work for you?

you are now hitting to different errors so wondering if thats related with latest storageclass fixes or not so can you please try with above mentioned older release.

dolesec commented 2 weeks ago

@kguptasangoma - I had to revert to version 17.0.2.18 in order to backup successfully, it breaks with version 17.0.2.20

msanthosh18 commented 2 weeks ago

Hi @dolesec Please update again module filestore 17.0.2.28, and give a try by mentioning Path for directory for the s3 bucket, as we also noticed such issue when we kept path as empty or slash Once we add directory name the backup was successful for us.

Also
Exmple :

dolesec commented 1 week ago

unfortunately, that's not working @kguptasangoma

i tried with the / and //

what does work as expected is 17.0.2.18

can we not revert until this is figured out?

Akarsh04 commented 1 week ago

Hi @dolesec It is possible that specifying just "/" for the storage path worked with filestore version 17.0.2.18. However, after the update to version 17.0.2.19, the libraries were updated to their latest versions. Specifically, league/flysystem-aws-s3-v3 was updated from version 1.0.30 to 3.28.0. After this update, it was observed that backup files will not be stored directly in the S3 bucket if the storage path is set to "/". However, if a directory path inside the bucket is specified, the backup will work without any issues. Note: The directory path specified in the filestore path must already exist in the S3 bucket.

Ex: If backup-17 is the directory which exists in our s3 bucket then path in filestore instance should be "backup-17/".

dolesec commented 1 week ago

I was able to make this work however it still feels like a regression and should be investigated ...

the only method I can use to make this work is to disable the option "Append Backup Name as a directory to the Storage path" ... currently I have one bucket defined for PBX backups and utilizing the append option above multiple PBX could share the same bucket and have subdirectories created for individual respective backups of numerous systems ... this still works on version 16 and through revision 17.0.2.18

if the append backup name causes this to break it should be removed or the module corrected such that this works as it does today on version 16

jissphilip commented 1 week ago

@dolesec filestore version 17.0.2.30

FreePBX / issue-tracker