Able to browse contents (enable directory indexing) of download.zfsonlinux.org

mailinglists35 commented 3 years ago

Describe the feature would like to see added to OpenZFS

I would like to browse the contents of download.zfsonlinux.org

How will this feature improve OpenZFS?

this will not improve openzfs, it will help me outsmart yum instead when I need to quickly look at some /epel/7.7/x86_64/ path

Additional context

curl/GET receive 403, browser receives garbage

mailinglists35 commented 3 years ago

hello? is it really that difficult to enable directory indexing for direct contents browsing/downloading, just like every linux repository in the world?

gmelikov commented 3 years ago

At least it's not a one-liner, iirc it hosts via aws s3, it doesn't have such functionality bult-in, but there's a workaround like https://github.com/rufuspollock/s3-bucket-listing

ruixingw commented 3 years ago

if possible can you also add rsync support to allow repo mirroring? I believe many open source mirror sites will be glad to maintain a mirror, and you can redirect users to these mirror sites to a). reduce the load on your main server and b). to allow users have better downloading experience.

bghira commented 3 years ago

you can use cmdline s3 tools to browse it.

mailinglists35 commented 3 years ago

you can use cmdline s3 tools to browse it.

that is only a hackish workaround.

the root question remains: why others allow browsing their repos with a web browser and openzfs doesn't?

gmelikov commented 3 years ago

the root question remains: why others allow browsing their repos with a web browser and openzfs doesn't?

Because this repo hosts via S3, and it doesn't have such functionality builtin, so there is not such question as "why listing is disabled".

hello? is it really that difficult to enable directory indexing for direct contents browsing/downloading, just like every linux repository in the world?

Yes, it's not just "enable some config param", AFAIK there is no such option in AWS S3.

bghira commented 3 years ago

you can use cmdline s3 tools to browse it.

that is only a hackish workaround.

hackish? i think trying to implement directory listing on S3 would be worse.

the root question remains: why others allow browsing their repos with a web browser and openzfs doesn't?

this is a question you could have searched for yourself. https://aws.amazon.com/blogs/compute/implementing-default-directory-indexes-in-amazon-s3-backed-amazon-cloudfront-origins-using-lambdaedge/

it's not easy! it's not even necessary. why do a lot of work for unnecessary gain? they're already paying for the bandwidth which is undoubtedly pretty costly.

ruixingw commented 3 years ago

you can use cmdline s3 tools to browse it.

I had a look at s3 cli tools. it seems the aws s3 ls command needs credentials for the bucket. Do you know if one can browse the ZFS s3 bucket anonymously? once all the file names are listed, http can be again used for downloading.

mskarbek commented 3 years ago

@ruixingw you can use reposync to mirror:

# dnf -qy install yum-utils
# cd $(mktemp -d)
# reposync -p ./ --download-metadata --repo=zfs
# reposync -p ./ --download-metadata --repo=zfs-kmod
# tree
.
├── zfs
│   ├── debug
│   │   ├── libnvpair3-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   ├── libuutil3-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   ├── libzfs4-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   ├── libzpool4-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   ├── zfs-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   └── zfs-test-debuginfo-2.0.5-1.el8.x86_64.rpm
│   ├── libnvpair3-2.0.5-1.el8.x86_64.rpm
│   ├── libuutil3-2.0.5-1.el8.x86_64.rpm
│   ├── libzfs4-2.0.5-1.el8.x86_64.rpm
│   ├── libzfs4-devel-2.0.5-1.el8.x86_64.rpm
│   ├── libzpool4-2.0.5-1.el8.x86_64.rpm
│   ├── python3-pyzfs-2.0.5-1.el8.noarch.rpm
│   ├── repodata
│   │   ├── 02c9d8fadf31aa8bfc5c082b48068c567cba12713acee5fc02186895accdb6a2-primary.sqlite.bz2
│   │   ├── 0fb8bb297fe0c16dfb263c3a37711b2a03977a5cf46667da1989ac96e43f1970-other.sqlite.bz2
│   │   ├── 2d25ea3d789f4ffb3cc3ed80bb506aa31c229ca4850c6d516f766d787559f3ff-filelists.sqlite.bz2
│   │   ├── 33010bfddb518956c91f68924a9a684a6ca1c8626afaf8cd35d3512674648b4b-other.xml.gz
│   │   ├── 6461405211728567ca10a8cba475c866b4f4b19b6348c5fb3e998bd9da9a2bb0-primary.xml.gz
│   │   ├── 75eb05717b84044387aa9a2566e41c92432e4219169c8988be79a6b4904bbb28-filelists.xml.gz
│   │   └── repomd.xml
│   ├── zfs-2.0.5-1.el8.x86_64.rpm
│   ├── zfs-debugsource-2.0.5-1.el8.x86_64.rpm
│   ├── zfs-dkms-2.0.5-1.el8.noarch.rpm
│   ├── zfs-dracut-2.0.5-1.el8.noarch.rpm
│   └── zfs-test-2.0.5-1.el8.x86_64.rpm
└── zfs-kmod
    ├── debug
    │   ├── kmod-zfs-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── libnvpair3-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── libuutil3-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── libzfs4-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── libzpool4-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── zfs-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   └── zfs-test-debuginfo-2.0.5-1.el8.x86_64.rpm
    ├── kmod-zfs-2.0.5-1.el8.x86_64.rpm
    ├── kmod-zfs-devel-2.0.5-1.el8.x86_64.rpm
    ├── libnvpair3-2.0.5-1.el8.x86_64.rpm
    ├── libuutil3-2.0.5-1.el8.x86_64.rpm
    ├── libzfs4-2.0.5-1.el8.x86_64.rpm
    ├── libzfs4-devel-2.0.5-1.el8.x86_64.rpm
    ├── libzpool4-2.0.5-1.el8.x86_64.rpm
    ├── python3-pyzfs-2.0.5-1.el8.noarch.rpm
    ├── repodata
    │   ├── 1437e00cf044c8e9726b625e1275111377016496639654a66bd01c1c13fe0977-filelists.sqlite.bz2
    │   ├── 4cee20430f8a5e2f8c18398e76dc66b5d7c33564b0e722b8f28ac46fe04190c3-primary.sqlite.bz2
    │   ├── 67f4ecb2d44051fe6a78d4e9b07c8979765ecf502aac56a1f6818bd35eae4bcf-other.sqlite.bz2
    │   ├── 798178caf4728f0fd959c91cc2091605317778880e2749d351a11b201576dcc8-filelists.xml.gz
    │   ├── 896e6043dc4dadd316350754543b71c100275fb0a53c8253ee2832b61189051d-other.xml.gz
    │   ├── c01d58e0476975e6e6725e619865737ef0f477f873578bbabbf2530c4556e27d-primary.xml.gz
    │   └── repomd.xml
    ├── zfs-2.0.5-1.el8.x86_64.rpm
    ├── zfs-debugsource-2.0.5-1.el8.x86_64.rpm
    ├── zfs-dracut-2.0.5-1.el8.noarch.rpm
    ├── zfs-kmod-debugsource-2.0.5-1.el8.x86_64.rpm
    └── zfs-test-2.0.5-1.el8.x86_64.rpm

6 directories, 51 files

Ofc you have to have repo installed on your host, otherwise reposync will fail.

ruixingw commented 3 years ago

@ruixingw you can use reposync to mirror:

# dnf -qy install yum-utils
# cd $(mktemp -d)
# reposync -p ./ --download-metadata --repo=zfs
# reposync -p ./ --download-metadata --repo=zfs-kmod
# tree
.
├── zfs
│   ├── debug
│   │   ├── libnvpair3-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   ├── libuutil3-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   ├── libzfs4-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   ├── libzpool4-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   ├── zfs-debuginfo-2.0.5-1.el8.x86_64.rpm
│   │   └── zfs-test-debuginfo-2.0.5-1.el8.x86_64.rpm
│   ├── libnvpair3-2.0.5-1.el8.x86_64.rpm
│   ├── libuutil3-2.0.5-1.el8.x86_64.rpm
│   ├── libzfs4-2.0.5-1.el8.x86_64.rpm
│   ├── libzfs4-devel-2.0.5-1.el8.x86_64.rpm
│   ├── libzpool4-2.0.5-1.el8.x86_64.rpm
│   ├── python3-pyzfs-2.0.5-1.el8.noarch.rpm
│   ├── repodata
│   │   ├── 02c9d8fadf31aa8bfc5c082b48068c567cba12713acee5fc02186895accdb6a2-primary.sqlite.bz2
│   │   ├── 0fb8bb297fe0c16dfb263c3a37711b2a03977a5cf46667da1989ac96e43f1970-other.sqlite.bz2
│   │   ├── 2d25ea3d789f4ffb3cc3ed80bb506aa31c229ca4850c6d516f766d787559f3ff-filelists.sqlite.bz2
│   │   ├── 33010bfddb518956c91f68924a9a684a6ca1c8626afaf8cd35d3512674648b4b-other.xml.gz
│   │   ├── 6461405211728567ca10a8cba475c866b4f4b19b6348c5fb3e998bd9da9a2bb0-primary.xml.gz
│   │   ├── 75eb05717b84044387aa9a2566e41c92432e4219169c8988be79a6b4904bbb28-filelists.xml.gz
│   │   └── repomd.xml
│   ├── zfs-2.0.5-1.el8.x86_64.rpm
│   ├── zfs-debugsource-2.0.5-1.el8.x86_64.rpm
│   ├── zfs-dkms-2.0.5-1.el8.noarch.rpm
│   ├── zfs-dracut-2.0.5-1.el8.noarch.rpm
│   └── zfs-test-2.0.5-1.el8.x86_64.rpm
└── zfs-kmod
    ├── debug
    │   ├── kmod-zfs-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── libnvpair3-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── libuutil3-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── libzfs4-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── libzpool4-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   ├── zfs-debuginfo-2.0.5-1.el8.x86_64.rpm
    │   └── zfs-test-debuginfo-2.0.5-1.el8.x86_64.rpm
    ├── kmod-zfs-2.0.5-1.el8.x86_64.rpm
    ├── kmod-zfs-devel-2.0.5-1.el8.x86_64.rpm
    ├── libnvpair3-2.0.5-1.el8.x86_64.rpm
    ├── libuutil3-2.0.5-1.el8.x86_64.rpm
    ├── libzfs4-2.0.5-1.el8.x86_64.rpm
    ├── libzfs4-devel-2.0.5-1.el8.x86_64.rpm
    ├── libzpool4-2.0.5-1.el8.x86_64.rpm
    ├── python3-pyzfs-2.0.5-1.el8.noarch.rpm
    ├── repodata
    │   ├── 1437e00cf044c8e9726b625e1275111377016496639654a66bd01c1c13fe0977-filelists.sqlite.bz2
    │   ├── 4cee20430f8a5e2f8c18398e76dc66b5d7c33564b0e722b8f28ac46fe04190c3-primary.sqlite.bz2
    │   ├── 67f4ecb2d44051fe6a78d4e9b07c8979765ecf502aac56a1f6818bd35eae4bcf-other.sqlite.bz2
    │   ├── 798178caf4728f0fd959c91cc2091605317778880e2749d351a11b201576dcc8-filelists.xml.gz
    │   ├── 896e6043dc4dadd316350754543b71c100275fb0a53c8253ee2832b61189051d-other.xml.gz
    │   ├── c01d58e0476975e6e6725e619865737ef0f477f873578bbabbf2530c4556e27d-primary.xml.gz
    │   └── repomd.xml
    ├── zfs-2.0.5-1.el8.x86_64.rpm
    ├── zfs-debugsource-2.0.5-1.el8.x86_64.rpm
    ├── zfs-dracut-2.0.5-1.el8.noarch.rpm
    ├── zfs-kmod-debugsource-2.0.5-1.el8.x86_64.rpm
    └── zfs-test-2.0.5-1.el8.x86_64.rpm

6 directories, 51 files

Ofc you have to have repo installed on your host, otherwise reposync will fail.

This is nice. Thanks!

openzfs / zfs