Account in URL

[sebastian-luna-valero]

Hello,

According to the docs, here is how to access a file in swift:

import fsspec

with fsspec.open("swift://server/account/container/object.txt", "r") as f:
    print(f.read())

Apologies for asking the obvious but, is account on the string above referring to a personal user account for the swift endpoint?

Do you still need account if you configure environment variables OS_STORAGE_URL and OS_AUTH_TOKEN for authentication?

Many thanks, Sebastian

[d70-t]

Hey @sebastian-luna-valero, thanks for asking! I'm sorry for the late response. The string above is referring to the account as given in the SWIFT Object Storage API.

It may or may not be a personal user account, but that likely depends on your particular swift setup. It's part of the OS_STORAGE_URL. The reason why swiftspec wants it in the path as well, is because swiftspec could handle multiple accounts at different servers (not via environment variables, but using manual confiuration). For this to work, it chooses the right auth token based on the server and account from the storage url.

[sebastian-luna-valero]

Thanks @d70-t

After reading https://docs.openstack.org/swift/latest/api/object_api_v1_overview.html I am trying to build

fsspec.open("swift://server/account/container/object.txt", "r")

Using:

• server: the url retured by openstack catalog show swift without the trailing /swift/v1 (since the original output is https://<server>/swift/v1)
• account: project ID or project name

However I am getting FileNotFoundError. I am sure the file exists.

I have also configured OS_AUTH_TOKEN and OS_STORAGE_URL with valid values, but no difference.

Am I doing something wrong?

[d70-t]

Hmm... I think I now get the problem. For the swift API, it's important to handle prefix (or server), account, container and object separately, because e.g. the ls request always has to go to the account/container API.

All these four components must be extracted from whatever is passed into fsspec.open("swift://..."). Fortunately account and container can never contain any /, but object may contain any amount of /s, thus we can't split by / from the back. What swiftspec currently does is to split by / from the front, assuming the prefix doesn't contain any further / and thus the prefix is actually only the server name.

In your case however, the prefix would be <server>/swift, which means that the swiftspec-heuristic for splitting the path fails. 😬

However, I don't yet know to improve on this. My current thought would be to add something like a path option to fsspec.open(). This would be backwards compatible, but would require in your case to write something like:

fsspec.open("swift://server/account/container/object.txt", "r", path="/swift")

Which is not super nice, but would allow to keep the above stated heuristics, but to construct prefix = server + path.