audeering / audbackend

Manage file storage on different backends
https://audeering.github.io/audbackend/
Other
3 stars 0 forks source link

FileSystem: support old file structure #142

Closed frankenjoe closed 1 year ago

frankenjoe commented 1 year ago

Closes https://github.com/audeering/audbackend/issues/136

Analogous to https://github.com/audeering/audbackend/pull/130 this adds FileSystem._use_legacy_file_structure() to support our legacy file systems on existing repositories.

Example

Default:

import tempfile
import audbackend
import audeer

with tempfile.TemporaryDirectory(dir='.') as host:

    backend = audbackend.create('file-system', host, 'repo')

    file = audeer.touch('file')
    backend.put_file(file, '/sub/file.txt', '1.0.0')
    backend.put_file(file, '/sub/file.tar.gz', '1.0.0')
    files = audeer.list_file_names(host, recursive=True, basenames=True)

print(files)
['repo/sub/1.0.0/file.tar.gz', 'repo/sub/1.0.0/file.txt']

Legacy:

import tempfile
import audbackend
import audeer

with tempfile.TemporaryDirectory(dir='.') as host:

    backend = audbackend.create('file-system', host, 'repo')
    backend._use_legacy_file_structure(extensions=['tar.gz'])

    file = audeer.touch('file')
    backend.put_file(file, '/sub/file.txt', '1.0.0')
    backend.put_file(file, '/sub/file.tar.gz', '1.0.0')
    files = audeer.list_file_names(host, recursive=True, basenames=True)

print(files)
['repo/sub/file/1.0.0/file-1.0.0.tar.gz', 'repo/sub/file/1.0.0/file-1.0.0.txt']
frankenjoe commented 1 year ago

@hagenw the doc job fails with:

image

I hope you had this before and know what do to do about it :)

codecov[bot] commented 1 year ago

Codecov Report

Merging #142 (275dec7) into dev (88080b9) will not change coverage. The diff coverage is 100.0%.

Files Changed Coverage Δ
audbackend/core/filesystem.py 100.0% <100.0%> (ø)
hagenw commented 1 year ago

I fixed the failing documentation test with https://github.com/audeering/audbackend/pull/143, now the Windows test is failing with

image

So, I guess you need to update something with the code/tests.

frankenjoe commented 1 year ago

So, I guess you need to update something with the code/tests.

done

hagenw commented 1 year ago

Great, thanks.