Open jmellor opened 8 years ago
repeatedly
commented
8 years ago We don't have a experience with mirroring tool so we don't have a good answer for this issue... Our repository is hosted on S3 and we use reprepro for creating repository. Does S3 or reprepro have a problem with mirroring tool?
jmellor
commented
8 years ago I think that the repository layout is incorrect. If I look at the distribution website, I can see:
packages.treasuredata.com/2/Ubuntu/trusty/dists/contrib
which is not correct. This layout means that there is prefix of 2/Ubuntu/trusty and a distribution called contrib and no component layer, so this is not a valid repository layout. The dists directory is currently at the wrong layer in the tree.
It should instead be:
packages.treasuredata.com/2/Ubuntu/dists/trusty/contrib
so that the distribution component would then be trusty and the component would be contrib.
Is it possible to fix this?
From: Masahiro Nakagawa [mailto:notifications@github.com] Sent: July-20-16 03:48 To: fluent/fluentd-docs Cc: John Mellor; Author Subject: Re: [fluent/fluentd-docs] Mirroring treasuredata fails (#235)
We don't have a experience with mirroring tool so we don't have a good answer for this issue... Our repository is hosted on S3 and we use reprepro for creating repository. Does S3 or reprepro have a problem with mirroring tool?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/fluent/fluentd-docs/issues/235#issuecomment-233870528, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKkBZaBRTO4Vm9_vw-O2aPoxNLsoqr9Iks5qXdKwgaJpZM4JPskY.
jmellor
commented
8 years ago I believe that there is an error either in the S3 bucket configuration, or in the web service that it is running.
I have attached a screenshot showing a firefox query for the Release file, along with the error response that appears to show that the Amazon site is misconfigured.
Is the repository site not public, and therefore only works for yourself or something simple like that?
From: Masahiro Nakagawa [mailto:notifications@github.com] Sent: July-20-16 03:48 To: fluent/fluentd-docs Cc: John Mellor; Author Subject: Re: [fluent/fluentd-docs] Mirroring treasuredata fails (#235)
We don't have a experience with mirroring tool so we don't have a good answer for this issue... Our repository is hosted on S3 and we use reprepro for creating repository. Does S3 or reprepro have a problem with mirroring tool?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/fluent/fluentd-docs/issues/235#issuecomment-233870528, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AKkBZaBRTO4Vm9_vw-O2aPoxNLsoqr9Iks5qXdKwgaJpZM4JPskY.
jmellor
commented
8 years ago (sorry, it looks like the github mail service is losing images. Here is the error image:
yuvraj2987
commented
7 years ago I am seeing this issue for macOS agent as well.
repeatedly
commented
7 years ago @yuvraj2987 Sorry, we accidentally revoke permission for this browser. We restore our setting and you now see the repository.
silverbadge
commented
6 years ago Hi,
We use Artifactory to proxy/cache remote repositories. We're getting this the same issue:
Unable to load bucket: packages.treasuredata.com
Are you able to help with this?
fujimotos
commented
6 years ago I'm looking into this issue and poking around for few minutes.
Apparently, the web app (S3 Browser) is just a directory viewer and not designed to proxy files. For actual contents of files, we need to access to the S3 endpoint directly.
In other words, we cannot retrieve a file from:
Instead, we need to access to the following URL:
This observation leads us to the following script (I modified the command posted by @jmellor):
debmirror --no-source --no-check-gpg --rsync-extra=none --progress --nocleanup \
--method=https --disable-ssl-verification \
-h packages.treasuredata.com.s3.amazonaws.com \
-a amd64 -s contrib -d trusty -r /2/ubuntu/trusty \
/path/to/designed/dir/This should properly mirror the remote files to you local directory.
(Beware the "--disable-ssl-verification" option in the script, though. We need this opt because Amazon does not provide us a proper SSL certificate for S3 endpoints; So it is not fully secure)
chicofranchico
commented
6 years ago For those using Aptly, you can use the following URL e.g. (for xenial v3)
http://packages.treasuredata.com.s3.amazonaws.com/3/ubuntu/xenial/ xenial contrib
And then:
aptly mirror create -architectures=amd64 <mirror_name> http://packages.treasuredata.com.s3.amazonaws.com/3/ubuntu/xenial/ xenial contrib
Help! I am unsuccessfully attempting to construct an internal mirror of the Ubuntu treasuredata repository using debmirror on a trusty machine that mirrors many other repositories, and suspecting that there is something wrong with the debian repository site.
The call to debmirror looks like:
which results in the following error log:
I've narrowed the problem down to:
This looks like it might be some kind of proxy error of the server-side. Is it possible to identify what the problem is, and how to get around it? Should I be using some other mirror tool that gets around this?