Open vadim opened 1 year ago
I am encountering this error too. I believe the issue is that the .cp
command is treating a file like a remote resource but it is in fact a local resource.
def _path(self, src):
src = parse.urlparse(str(src))
if src.scheme and src.scheme not in self.__ALLOWED_SCHEMES:
raise ValueError(
f'unsupported scheme "{src.scheme}". DBUtils in the SDK only supports local, root DBFS, and '
'UC Volumes paths, not external locations or DBFS mount points.')
if src.scheme == 'file':
return _LocalPath(src.geturl())
if src.path.startswith('/Volumes'):
return _VolumesPath(self._files_api, src.geturl())
return _DbfsPath(self._dbfs_api, src.geturl())
I believe it might be improperly handling the case where the scheme comes back as ''
. It will then automatically treat it as a dbfs
file instead of a local file.
One workaround right now would be to create the local file path as a uri
(if using a Path
object, call the .as_uri()
method on it before passing it in as a string. This will append the file
schema so it appears as a local file object since urllib
will parse it correctly.
EDIT:
Maybe another way to look at this, the __ALLOWED_SCHEMES
don't account for situations in which the scheme is an empty string ''
, which happens when you pass a path like /users/myuser/desktop
for example.
__ALLOWED_SCHEMES = [None, 'file', 'dbfs']
Description Lack of parity and unexpected behavior using
w.dbutils.fs.cp()
Reproduction
Expected behavior I expect
w.dbutils.fs.cp()
to behave like the databricks CLI commanddatabricks fs cp src dest
.Debug Logs
Other Information
Additional context N/A