fsspec / universal_pathlib

pathlib api extended to use fsspec backends
MIT License
211 stars 36 forks source link

UPath formatting of empty netloc breaks compatibility #151

Closed tehunter closed 5 months ago

tehunter commented 9 months ago

UPath._format_parsed_parts currently drops the netloc portion completely when netloc is empty. This leads to a shortened URI with single slash format.

For example, _format_parsed_parts currently shortens protocol:///path to protocol:/path. This unfortunately breaks compatibility with RFC 3986 format, which is assumed by fsspec and third-party libraries that use it (e.g. pandas). Per RFC 3986, an empty host/authority field can be valid.

UPath should offer an attribute to allow implementations to specify whether empty netloc's are valid or not. In the case where they are valid, it shouldn't treat empty string netloc any differently. Without a flag like this, implementations would need to override _format_parsed_parts to change one line.

I am building a UPath implementation for the boxfs implementation and I have to do just that. Netloc (URI host/authority) doesn't have a meaning in this "protocol", so I plan to leave it blank.

ap-- commented 9 months ago

Hi @tehunter

Thanks for opening the issue!

Your suggestion definitely sounds like a useful addition. At some point it would also make sense to normalize upaths URIs to the at-least-2-slash versions by default to be better compatible with fsspec. Although I need to give this a bit more thought together with relative URIs.

My current understanding of RFC 3986 is that protocol:/path uri is within RFC 3986 spec and equivalent to protocol:///path unless we are provided with a specific RFC for the protocol (scheme) (see: rfc3986 Section 3: where hier-part would be path-absolute or below)

For now I would say we should add:

class UPath(...):
    _uri_keep_empty_authority: bool = False

Would you want to implement a PR with the changes? (I'm currently working on python 3.12 support)

Cheers, Andreas :smiley: