Error listing backups in backup store

[the-so6]

Hi Lirt,

Thanks for your reactivity regarding our previous issue. By the way, we are facing an other problem. We cannot restore from a completed backup. We have this error all the time into logs :

time="2021-11-19T14:41:46Z" level=info msg="ObjectStore.Init called" cmd=/plugins/velero-plugin-swift controller=backup-sync logSource="/go/src/github.com/Lirt/velero-plugin-swift/src/swift/object_store.go:31" pluginName=velero-plugin-swift
time="2021-11-19T14:41:46Z" level=info msg="Trying to authenticate against Openstack using environment variables (including application credentials) or using files ~/.config/openstack/clouds.yaml, /etc/openstack/clouds.yaml and ./clouds.yaml" cmd=/plugins/velero-plugin-swift controller=backup-sync logSource="/go/src/github.com/Lirt/velero-plugin-swift/src/utils/auth.go:50" pluginName=velero-plugin-swift
time="2021-11-19T14:41:46Z" level=info msg="Authentication successful" cmd=/plugins/velero-plugin-swift controller=backup-sync logSource="/go/src/github.com/Lirt/velero-plugin-swift/src/utils/auth.go:70" pluginName=velero-plugin-swift
time="2021-11-19T14:41:46Z" level=info msg=ListCommonPrefixes bucket=velero-test cmd=/plugins/velero-plugin-swift controller=backup-sync delimiter=/ logSource="/go/src/github.com/Lirt/velero-plugin-swift/src/swift/object_store.go:117" pluginName=velero-plugin-swift prefix=backups/
time="2021-11-19T14:41:46Z" level=error msg="Error listing backups in backup store" backupLocation=default controller=backup-sync error="rpc error: code = Unknown desc = failed to list objects in bucket velero-test: invalid character 'b' looking for beginning of value" logSource="pkg/controller/backup_sync_controller.go:182"

The b character is the first letter from the prefix (backups), we verified that by changing the prefix. This error appears all the time but is not blocking the backup.

We are using velero 1.7.0 and tried with 1.6.3 without restic. It seems that is blocking the restore, this process seems very long (more than 10min for one PV)

Do not hesitate if you need further information

Thanks in advance

[the-so6]

Hi,

We have added some print to find an explanation. code added at : object_store.go / ListCommonPrefixes ()

func (o *ObjectStore) ListCommonPrefixes(bucket string, prefix string, delimiter string) ([]string, error) {
    log := o.log.WithFields(logrus.Fields{
        "bucket":    bucket,
        "delimiter": delimiter,
        "prefix":    prefix,
    })
    log.Infof("ListCommonPrefixes")

    opts := objects.ListOpts{
        Prefix:    prefix,
        Delimiter: delimiter,
        Full:      true,
    }

    log.Infof("%+v", opts)

    pager := objects.List(o.client, bucket, opts)

    log.Infof("%+v", pager)

    allPages, err := pager.AllPages()
    if err != nil {
        return nil, fmt.Errorf("failed to list objects in bucket %v: %v", bucket, err)
    }

and this is the result :

time="2021-11-22T11:07:39Z" level=info msg="{Full:true Limit:0 Marker: EndMarker: Format: Prefix:backups/ Delimiter:/ Path:}" bucket=velero-test cmd=/plugins/velero-plugin-swift controller=backup-sync delimiter=/ logSource="/go/src/github.com/Lirt/velero-plugin-swift/src/swift/object_store.go:125" pluginName=velero-plugin-swift prefix=backups/
time="2021-11-22T11:07:39Z" level=info msg="{client:0xc000184820 initialURL:https://swift.redacted.com/v1/AUTH_xxxxxxxxxx/velero-test?delimiter=%2F&prefix=backups%2F createPage:0xb2a920 firstPage:<nil> Err:<nil> Headers:map[Accept:application/json Content-Type:application/json]}" bucket=velero-test cmd=/plugins/velero-plugin-swift controller=backup-sync delimiter=/ logSource="/go/src/github.com/Lirt/velero-plugin-swift/src/swift/object_store.go:129" pluginName=velero-plugin-swift prefix=backups

It looks good but the error is still present. Do you see something wrong or the problem can come from our provider ?

[Lirt]

Hi @the-so6,

I have some questions.

1. Does your backupstoragelocation looks something like this?

spec:
  objectStorage:
    bucket: <BUCKET_NAME>
    prefix: backups/
  provider: community.openstack.org/openstack

2. Do you mean that this occurs only when using restic? Does backup and restore with prefix works well in your case without restic?

Delimiter and Prefix looks to be set correctly.

failed to list objects in bucket velero-test: invalid character 'b' looking for beginning of value

I don't understand this error message though. This happens after the List+AllPages function. Quick search says this is error when JSON unmarshall is done on non-JSON response.

I will try to replicate this and troubleshoot what is wrong in this case. Maybe it's a bug in gophercloud library, but I would need some time to check that.

[the-so6]

HI @Lirt ,

this is our backupstoragelocation

  objectStorage:
    bucket: velero-test
  provider: community.openstack.org/openstack

I do not set the prefix here because if I did I would have that behavior : prefix=backups/backups/

We want to try to do not used restic at all so no we don't install it. We are trying to find a solution too, I keep you inform if we found a solution or a workaround.

Thanks

[nodefourtytwo]

I don't understand this error message though. This happens after the List+AllPages function. Quick search says this is error when JSON unmarshall is done on non-JSON response.

Could this mean our cloud provider gives a non-JSON response to the call to this URL : https://swift.redacted.com/v1/AUTH_xxxxxxxxxx/velero-test?delimiter=%2F&prefix=backups%2F ? Do you have any idea how we could test that?

That would be strange that our provider's Swift behave differently than others.

Thanks for your help.

[Lirt]

Could this mean our cloud provider gives a non-JSON response to the call to this URL

It is very unlikely IMO.

Here are good examples to test directory listing with prefixes and delimiters - https://docs.openstack.org/swift/latest/api/pseudo-hierarchical-folders-directories.html

You need to get the auth token and it should be easy with swift python CLI swift auth.

For the rest just replace your Swift URL, container (bucket) name, prefix and delimiter.

Let me know what have you found.

[Lirt]

I use SwiftStack for all testing of this plugin.

From my test with URL https://<URL>/v1/<USER>/<CONTAINER>\?delimiter\=/\?prefix\=backups/:

1. If I don't specify header Accept: application/json I receive response in plain.
2. If I specify header Accept: application/json I receive response in json.

As we see in your case JSON headers are set in your case (becuase we use Full - check the gophercloud code), but we need to see from manual test if your Swift does the same as mine and if you really receive valid parsable JSON response in the Body.

If nothing helps, we can also try to force using JSON Format in ListOpts in the code.

[the-so6]

HI @Lirt, We made some test regarding swift output, and this is what we got:

curl -X GET -i -H "Accept: application/json" -H "Content-Type:application/json" -H "X-Auth-Token: $token" "https://swift.redacted/v1/AUTH_xxxxx/velero-test?delimiter=%2F&prefix=backups%2F"
HTTP/1.1 200 OK
X-Trans-Id: tx08170000017d4c5d93bf5-40830baef0
Accept-Ranges: bytes
Content-Length: 731
Content-Type: application/json;charset=UTF-8
Date: Tue, 23 Nov 2021 10:35:40 GMT
X-Container-Bytes-Used: 32562966
X-Container-Object-Count: 134
X-Storage-Policy: Policy-0
X-Timestamp: 1637078989.18700

[{"subdir": backups/xxxxx-minion/}, {"subdir": backups/athena-backup-20211119-043556/}, {"subdir": backups/athena-backup-20211120-060535/}, {"subdir": backups/athena-backup-20211121-060535/}, {"subdir": backups/athena-backup-20211122-060541/}, {"subdir": backups/athena-backup-20211123-060437/}, {"subdir": backups/athena-fdsf-backup-20211118-080537/}, {"subdir": backups/athena-erfwr-backup-20211118-104720/}, {"subdir": backups/athena-fgfg-backup-test-2-20211118-112403/}, {"subdir": backups/athena-werer-backup-test-2-20211118-141315/}, {"subdir": backups/docs/}, {"subdir": backups/docs2/}, {"subdir": backups/keycloak/}, {"subdir": backups/redis-bkp/}, {"subdir": backups/redis-wout-annotation/}, {"subdir": backups/redis/}]

The output is not a valid json. the " are missing around subdir paths. Do you have the same behavior on your swiftstack?

Thanks

[Lirt]

Hi again,

In my case it is quoted properly.

I do the request same way as you do and result is this (I created test directories to simulate similar answer):

curl \
  -X GET \
  -i \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -H "X-Auth-Token: $OS_AUTH_TOKEN" \
  https://<URL>/v1/<AUTH_>/<CONTAINER>\?delimiter\=/\&prefix\=test/

HTTP/1.1 200 OK
Content-Length: 89
X-Container-Object-Count: 62
X-Timestamp: 1637240619.47921
Accept-Ranges: bytes
X-Storage-Policy: Standard-Replica
Last-Modified: Tue, 23 Nov 2021 11:00:22 GMT
X-Container-Bytes-Used: 14149268
Content-Type: application/json; charset=utf-8
Vary: Accept
X-Trans-Id: ...
X-Openstack-Request-Id: ...
Date: Tue, 23 Nov 2021 11:01:03 GMT

[{"subdir": "test/dir-one/"}, {"subdir": "test/dir-three/"}, {"subdir": "test/dir-two/"}]

[Lirt]

Can you also try to append ?format=json to the request params? Just to check if something changes.

[the-so6]

yes, we tried and the result is still the same :cry: We open an issue on OTC to try to understand why their answer is not a correct json

[the-so6]

Hi @Lirt,

We double check with OTC support and Swift is no longer support. We need to use s3 instead. You can close this issue. Thanks again for your help

[Lirt]

Hi @the-so6,

I'm sorry to hear that.

Thank you for reporting anyway. It can be useful for other people in OTC.