chore: update Python version

igalic commented 4 years ago

the default is now 3.7 on FreeBSD.

igalic commented 4 years ago

Travis error is unrelated:

./libioc/helpers_ioctl.py:44:23: T484 No overload variant of "ioctl" matches argument types "int", "Any", "bytes"
./libioc/helpers_ioctl.py:44:23: T400 note: Possible overload variants:
./libioc/helpers_ioctl.py:44:23: T400 note: def ioctl(__fd, Union[int, HasFileno], int, int = ..., bool = ...) -> int
./libioc/helpers_ioctl.py:44:23: T400 note: def ioctl(__fd, Union[int, HasFileno], int, Union[bytearray, memoryview, array[Any]], Literal[True] = ...) -> int
./libioc/helpers_ioctl.py:44:23: T400 note: <2 more non-matching overloads not shown>

just means newer mypy version is now detecting better… things?

igalic commented 4 years ago

Test run is failing, but i don't know why

``` env ZPOOL="ioc-test-`uname -r`" make test pytest tests --zpool ioc-test-12.1-RELEASE ============================= test session starts ============================== platform freebsd12 -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /usr/local/bin/python3.7 cachedir: .pytest_cache rootdir: /tmp/cirrus-ci-build, inifile: setup.cfg plugins: cov-2.8.1, pep8-1.0.6, mock-3.0.0 collecting ... collected 63 items tests/test_Backup.py::TestBackup::test_nullfs_basejail_can_be_exported_standalone ERROR [ 1%] ==================================== ERRORS ==================================== _ ERROR at setup of TestBackup.test_nullfs_basejail_can_be_exported_standalone _ release = root_dataset = zfs = @pytest.fixture def local_release( release: 'libioc.Release.ReleaseGenerator', root_dataset: libzfs.ZFSDataset, zfs: libzfs.ZFS ) -> 'libioc.Release.ReleaseGenerator': """Mock a local release.""" if not release.fetched: > release.fetch(fetch_updates=True, update=True) tests/conftest.py:181: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tests/libioc/Release.py:1087: in fetch event_scope=event_scope tests/libioc/Release.py:742: in fetch self._extract_assets() tests/libioc/Release.py:969: in _extract_assets destination=self.root_dir, tests/libioc/Release.py:370: in root_dir if self.root_dataset.mountpoint: tests/libioc/LaunchableResource.py:131: in root_dataset self._require_dataset_mounted(root_dataset) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = dataset = def _require_dataset_mounted(self, dataset: libzfs.ZFSDataset) -> None: if dataset.mountpoint is None: raise libioc.errors.DatasetNotMounted( dataset=dataset, > logger=self.logger ) E libioc.errors.DatasetNotMounted tests/libioc/LaunchableResource.py:118: DatasetNotMounted ---------------------------- Captured stdout setup ----------------------------- Log directory '/var/log/iocage' created Cache miss: .cache/libioc/download.freebsd.org/ftp/releases/amd64/amd64/12.1-RELEASE/base.txz (http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/base.txz) http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/base.txz saved to .cache/libioc/download.freebsd.org/ftp/releases/amd64/amd64/12.1-RELEASE/base.txz Cache miss: .cache/libioc/download.freebsd.org/ftp/releases/amd64/amd64/12.1-RELEASE/lib32.txz (http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/lib32.txz) http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/lib32.txz saved to .cache/libioc/download.freebsd.org/ftp/releases/amd64/amd64/12.1-RELEASE/lib32.txz Cache miss: .cache/libioc/download.freebsd.org/ftp/releases/amd64/amd64/12.1-RELEASE/MANIFEST (http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/MANIFEST) http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/MANIFEST saved to .cache/libioc/download.freebsd.org/ftp/releases/amd64/amd64/12.1-RELEASE/MANIFEST Dataset 'ioc-test-12.1-RELEASE/libioc-test/releases/12.1-RELEASE/root' is not mounted ---------------------------- Captured stderr setup ----------------------------- 127.0.0.1 - - [07/Apr/2020 13:46:14] "GET http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/base.txz HTTP/1.1" 200 - 127.0.0.1 - - [07/Apr/2020 13:46:20] "GET http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/lib32.txz HTTP/1.1" 200 - 127.0.0.1 - - [07/Apr/2020 13:46:21] "GET http://download.FreeBSD.org/ftp/releases/amd64/amd64/12.1-RELEASE/MANIFEST HTTP/1.1" 200 - !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!! ========================= 1 error in 89.78s (0:01:29) ========================== /usr/local/lib/python3.7/site-packages/pep8.py:110: FutureWarning: Possible nested set at position 1 EXTRANEOUS_WHITESPACE_REGEX = re.compile(r'[[({] | []}),;:]') *** Error code 1 Stop. make: stopped in /tmp/cirrus-ci-build ```

yonk42 commented 4 years ago

A couple of days ago I upgraded my server to FreeBSD 12.1-RELEASE and used this branch to upgrade my years old libiocage to libioc + ioc. I used this branch to install and am now encountering the same issue as you do in your test run, namely a "Dataset xyz is not mounted". It occurs when creating a new jail and also when fetching a new release:

# ioc fetch -r 12.1-RELEASE
[-] FetchRelease@12.1-RELEASE: ...
  [+] ReleasePrepareStorage@12.1-RELEASE: OK [0.112s]
  [+] ReleaseDownload@12.1-RELEASE: OK [6.487s]
    [+] ReleaseAssetDownload@12.1-RELEASE: OK [2.291s]
  [-] ReleaseExtraction@12.1-RELEASE: FAILED [0.714s]
Dataset 'zroot/iocage/releases/12.1-RELEASE/root' is not mounted

Altough the dataset is actually mounted:

# mount | grep 'zroot/iocage/releases/12.1-RELEASE/root'
zroot/iocage/releases/12.1-RELEASE/root on /iocage/releases/12.1-RELEASE/root (zfs, local, noatime, nfsv4acls)

Digging through the codebase, I traced the issue back to the dataset.mountpoint call in _require_dataset_mounted (L115 in LaunchableResource.py), but I cannot reproduce libzfs returing None for the dataset's mountpoint.

I added a sleep before the mountpoint is checked and ran a script in a loop to check the mountpoint using both libzfs and libioc.ZFS.get_zfs(), but both reported the mountpoint correctly every time. I have no idea what the issue is here, but since I am interested in a fix I thought I'd shar my findings here. If there is anything I can do (testing, etc.) I'd gladly help out. I can reproduce the issue in @gronke's chore/update-default-python-version branch as well. I also tried using the master branch of libzfs but that did not help either.

Test script for reference:

#!/usr/bin/env python3.7
import libzfs
import libioc

dataset_name = "zroot/iocage/releases/12.1-RELEASE"

stdzfs = libzfs.ZFS()
ioczfs = libioc.ZFS.get_zfs()

def check(zfs):
    normal = zfs.get_dataset(dataset_name)
    root = zfs.get_dataset(dataset_name + "/root")
    print(f"normal mountpoint: {normal.mountpoint}")
    print(f"root mountpoint: {root.mountpoint}")

print("without libioc zfs:")
check(stdzfs)
print("with libioc zfs:")
check(ioczfs)

BTW, shouldn't you also change the python version for the install-travis target to `$pyver$ as well?

igalic commented 4 years ago

BTW, shouldn't you also change the python version for the install-travis target to `$pyver$ as well?

well, that's probably a good idea (done)

urosgruber commented 4 years ago

I'm seeing the same problem with

Dataset 'zroot/iocage/releases/12.1-RELEASE/root' is not mounted

urosgruber commented 4 years ago

Also same problem creating new jails.

urosgruber commented 4 years ago

Not sure if it helps but I reverted py-libzfs from 20191113 to 20181220 and error seems to go away