search

bsdci / libioc

A Python library to manage jails with ioc{age,ell}
https://bsd.ci/libioc
Other
38 stars 11 forks source link

ioc fetch fails with 404 #624

Closed alice-wl closed 5 years ago

alice-wl commented 5 years ago

ioc, version 0.3.1 2018/10/04

a stable release.

[0] 9.3-RELEASE (EOL)
[1] 10.1-RELEASE (EOL)
[2] 10.2-RELEASE (EOL)
[3] 10.3-RELEASE (EOL)
[4] 10.4-RELEASE (EOL)
[5] 11.0-RELEASE (EOL)
[6] 11.1-RELEASE (EOL)
[7] 11.2-RELEASE
[8] 12.0-RELEASE

Type the number of the desired RELEASE
Press [Enter] to fetch the default selection (12.0-RELEASE) [8]:
[+] FetchRelease@12.0-RELEASE: already downloaded
[+] ReleaseConfiguration@12.0-RELEASE: OK [0.013s]
[-] ReleaseUpdatePull@12.0-RELEASE: FAILED [0.507s]
Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/ioc/fetch.py", line 141, in cli
    fetch_updates=fetch_updates
  File "/usr/local/lib/python3.6/site-packages/ioc/__init__.py", line 87, in print_events
    for event in generator:
  File "/usr/local/lib/python3.6/site-packages/iocage/Release.py", line 720, in fetch
    for event in self.updater.fetch(event_scope=_scope):
  File "/usr/local/lib/python3.6/site-packages/iocage/ResourceUpdater.py", line 277, in fetch
    self._pull_updater()
  File "/usr/local/lib/python3.6/site-packages/iocage/ResourceUpdater.py", line 244, in _pull_updater
    local=f"{self.host_updates_dir}/{self.update_conf_name}"
  File "/usr/local/lib/python3.6/site-packages/iocage/ResourceUpdater.py", line 220, in _download_updater_asset
    _request.urlretrieve(url, local)  # nosec: url validated
  File "/usr/local/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
Corvan commented 5 years ago

I do not have this problem:

lars@freebsd:/home/lars/PycharmProjects/iocage git:(features/destroy*) $ sudo ioc fetch
[0] 9.3-RELEASE (EOL)
[1] 10.1-RELEASE (EOL)
[2] 10.2-RELEASE (EOL)
[3] 10.3-RELEASE (EOL)
[4] 10.4-RELEASE (EOL)
[5] 11.0-RELEASE (EOL)
[6] 11.1-RELEASE (EOL)
[7] 11.2-RELEASE
[8] 12.0-RELEASE

Type the number of the desired RELEASE
Press [Enter] to fetch the default selection (12.0-RELEASE) [8]: 
[-] FetchRelease@12.0-RELEASE: ...
  [-] ReleasePrepareStorage@12.0-RELEASE: ...
  [+] ReleasePrepareStorage@12.0-RELEASE: OK [0.441s]
  [-] ReleaseDownload@12.0-RELEASE: ...
  [+] ReleaseDownload@12.0-RELEASE: OK [201.377s]

But that was with a new fetch not one onto an existing Release

Corvan commented 5 years ago

OK with an existing Release I have the same Problem, and even after ioc destroy -r 12.0-RELEASE after the fetch did not work again.

igalic commented 5 years ago

let's summarise:

that about right?

Corvan commented 5 years ago

yep

gronke commented 5 years ago

I've also tested on different systems. @alice-wl was so kind to show me the issue in an offline-discussion, so that I was able to track down the issue to different update assets after fetching the updates outside of the jail or within.

Could be related to https://github.com/bsdci/libioc/issues/621 where UNAME_R should be set to 12.0-RELEASE. I was not able to reproduce this issue on one of my systems, so steps to reproduce this on a clean system would be nice.

  • even if there already is a (specific) patch-update
  • ioc fetch should be idempotent

Yes, everything else bad!

  • ioc destroy -r must destroy all artefacts of a(n unused) release, including all snapshots created by the these update process

I also agree here, although release deletion only works under the circumstance that no basejail (which is the default) are created from it - otherwise the release cannot be destroyed because of the ZFS dataset relation. Promoting a jail does not make sense because then it would look that other jails depend on it, which is not true.

Corvan commented 5 years ago

I had the problem after playing around with iocage 1.1 first (but I thought I cleaned the jails and releases of it) and ioc afterwards. But what was strange was, that extracting the release after successully downloading it, took about 2h and the I killed it. Afterwarfs I tried to destroy the release (which did not lead to errors) and download it again. This download startet but failed after some time with 404. And with the partial download it failed right away. Destroying the partial download worked, but downloading it again lead to the 404 after some time again.

igalic commented 5 years ago

we extract using rsync, so having some unexpected structures might lead to very long (infinite) extraction time

gronke commented 5 years ago

But what was strange was, that extracting the release after successully downloading it, took about 2h and the I killed it.

I have no explanation for that yet. If it occurs another time, please copy the downloaded .txz file while the extraction hangs, so that we have a copy to reproduce the issue on. We're using tar.extractall, so so direct output of extracted files, we'd have to loop manually. Having a dump of the file causing issues, would be as informative. We cannot ignore this issue, because it could cause failure of the host system by filling up the disk, memory or CPU (not sure if it's a feature to only support one core for extraction 😞 )

Afterwarfs I tried to destroy the release (which did not lead to errors) and download it again.

Compatibility with existing releases is mandatory, because a release dataset that was cloned is a dependency of its jails and thus cannot be deleted. What we want is an idempotent ioc fetch that leaves us with a snapshot @p<N> of the latest fetched release. (For HardenedBSD N matches the patchlevel number.)

This download startet but failed after some time with 404

When running ioc -d spam fetch you should find the URL in your stdout. You can then investigate whether that URL is wrong or why you cannot download it. Error 404 is reported by the remote server, so either the URL is wrong, the remote is having trouble or there are caching/proxy issues on the way to the upstream server.

Corvan commented 5 years ago

When running ioc -d spam fetch you should find the URL in your stdout.

I'll try

I have no explanation for that yet. If it occurs another time, please copy the downloaded .txz file while the extraction hangs, so that we have a copy to reproduce the issue on. We're using tar.extractall, so so direct output of extracted files, we'd have to loop manually. Having a dump of the file causing issues, would be as informative.

I wanted to, but I was not able to, see the next problems I had. I will destroy the zpool and try from the top.

Corvan commented 5 years ago

So, now:

lars@freebsd:/home/lars/PycharmProjects/iocage git:(features/destroy*) $ sudo ioc -d spam fetch -r 12.0-RELEASE
/etc/rc.conf was read from /etc/rc.conf
Updated /etc/rc.conf data from /etc/rc.conf
Found active ZFS pool zdata
[-] FetchRelease@12.0-RELEASE: ...
  [-] ReleasePrepareStorage@12.0-RELEASE: ...
  [+] ReleasePrepareStorage@12.0-RELEASE: OK [0.7s]
  [-] ReleaseDownload@12.0-RELEASE: ...
Starting download of https://download.freebsd.org/ftp/releases/amd64/amd64/12.0-RELEASE/base.txz
https://download.freebsd.org/ftp/releases/amd64/amd64/12.0-RELEASE/base.txz was saved to /iocage/releases/12.0-RELEASE/base.txz
Starting download of https://download.freebsd.org/ftp/releases/amd64/amd64/12.0-RELEASE/lib32.txz
https://download.freebsd.org/ftp/releases/amd64/amd64/12.0-RELEASE/lib32.txz was saved to /iocage/releases/12.0-RELEASE/lib32.txz
  [+] ReleaseDownload@12.0-RELEASE: OK [315.164s]

  [-] ReleaseExtraction@12.0-RELEASE: ...
hashes have not yet been downloaded
Downloading hashes from https://download.freebsd.org/ftp/releases/amd64/amd64/12.0-RELEASE/MANIFEST
Hashes downloaded to /iocage/releases/12.0-RELEASE/MANIFEST
9 hashes read from /iocage/releases/12.0-RELEASE/MANIFEST
Asset base.txz has a valid signature (360df303fac75225416ccc0c32358333b90ebcd58e54d8a935a4e13f158d3465)
Verifying file structure in /iocage/releases/12.0-RELEASE/base.txz
Extracting /iocage/releases/12.0-RELEASE/base.txz
/iocage/releases/12.0-RELEASE/base.txz was extracted to /iocage/releases/12.0-RELEASE/root
Asset lib32.txz has a valid signature (6c3618dfdf001b3232101327a510f7c6b9e4ba58e59bd6509478e89b83da883b)
Verifying file structure in /iocage/releases/12.0-RELEASE/lib32.txz
Extracting /iocage/releases/12.0-RELEASE/lib32.txz
/iocage/releases/12.0-RELEASE/lib32.txz was extracted to /iocage/releases/12.0-RELEASE/root
  [+] ReleaseExtraction@12.0-RELEASE: OK [49.948s]

[+] FetchRelease@12.0-RELEASE: OK [365.812s]

[-] ReleaseConfiguration@12.0-RELEASE: ...
Updated /etc/rc.conf data from /iocage/releases/12.0-RELEASE/root/etc/rc.conf
Writing /etc/rc.conf to /iocage/releases/12.0-RELEASE/root/etc/rc.conf
  netif_enable="NO"
  sendmail_enable="NO"
  sendmail_submit_enable="NO"
  sendmail_msp_queue_enable="NO"
  sendmail_outbound_enable="NO"
  syslogd_flags="-ss"
/etc/sysctl.conf was read from /iocage/releases/12.0-RELEASE/root/etc/sysctl.conf
Updated /etc/sysctl.conf data from /iocage/releases/12.0-RELEASE/root/etc/sysctl.conf
Writing /etc/sysctl.conf to /iocage/releases/12.0-RELEASE/root/etc/sysctl.conf
  net.inet.ip.fw.enable="0"
[+] ReleaseConfiguration@12.0-RELEASE: OK [0.006s]

[-] ReleaseUpdatePull@12.0-RELEASE: ...
Downloading https://svn.freebsd.org/base/release/12.0.0/usr.sbin/freebsd-update/freebsd-update.sh
Update-asset usr.sbin/freebsd-update/freebsd-update.sh for release '12.0-RELEASE' saved to /iocage/releases/12.0-RELEASE/updates/freebsd-update.sh
Downloading https://svn.freebsd.org/base/release/12.0.0/etc/freebsd-update.conf
[-] ReleaseUpdatePull@12.0-RELEASE: FAILED [11.274s]

Traceback (most recent call last):
  File "/usr/local/bin/ioc", line 10, in <module>
    sys.dd:exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/ioc/fetch.py", line 141, in cli
    fetch_updates=fetch_updates
  File "/usr/local/lib/python3.6/site-packages/ioc/__init__.py", line 87, in print_events
    for event in generator:
  File "/usr/local/lib/python3.6/site-packages/iocage/Release.py", line 715, in fetch
    for event in self.updater.fetch(event_scope=_scope):
  File "/usr/local/lib/python3.6/site-packages/iocage/ResourceUpdater.py", line 276, in fetch
    self._pull_updater()
  File "/usr/local/lib/python3.6/site-packages/iocage/ResourceUpdater.py", line 243, in _pull_updater
    local=f"{self.host_updates_dir}/{self.update_conf_name}"
  File "/usr/local/lib/python3.6/site-packages/iocage/ResourceUpdater.py", line 219, in _download_updater_asset
    _request.urlretrieve(url, local)  # nosec: url validated
  File "/usr/local/lib/python3.6/urllib/request.py", line 248, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

Do you still need the downloaded images?

gronke commented 5 years ago

@Corvan the URL of freebsd-update.conf has changes in the sources. https://github.com/bsdci/libioc/pull/581 has patched this issue for releases >= 12.0. Can you make sure that you are using the latest version of ioc?

mkdir -p /usr/local/src
git clone https://github.com/bsdci/ioc /usr/local/src/ioc
cd /usr/local/src/ioc/
make install

# verify installed versions
cd /tmp
ioc version
python3.6 -c "import libioc; print(libioc.VERSION)"