search

plone / Installers-UnifiedInstaller

Linux/BSD/OSX Installer
28 stars 24 forks source link

5.2.3 overhaul, testing and GH actions #110

Closed jensens closed 3 years ago

jensens commented 4 years ago

GH actions ftw. And some overhaul of the scripts. This should do it for win, macos and Linux.

Build now runs on Linux and macOS only and is tested with python 3.8 only. This is sufficient. Installer run can be tested in a matrix on all 3 systems, using both builds, using all 4 supported python versions. Since windows tests were never supported this works for now only on unix, but I started a port of the tests to windows.

fun-fact: the installer tgz is available for instant download from the actions tab.

there is more to-do, in direction of real CI, but for a sunny Sunday this is enough for now.

TODO:

NICE TO HAVE/ OWN TASK

jensens commented 4 years ago

Nice, test are broken on macos/py3.7 - all other combinations are working.

tkimnguyen commented 3 years ago

The installer I downloaded from the Actions artifacts ran great on macOS 10.15.7 Catalina: Plone 5.2.2, Python 2.7.16 reported on the control panel.

One minor note: shouldn't it report Plone 5.2.3?

tkimnguyen commented 3 years ago

On same macOS 10.15.7, attempt to build using Python 3.8.2 fails:

$ python3
Python 3.8.2 (default, Oct  2 2020, 10:45:42) 
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D
$ which python3
/usr/bin/python3

$ ./install.sh --with-python=`which python3` --password=admin standalone

Testing /usr/bin/python3 for Zope/Plone requirements....
/usr/bin/python3 looks OK. We will use it.

Rootless install method chosen. Will install for use by system user kimadmin

Detailed installation log being written to /Users/kimadmin/Downloads/Plone-5.2.3-UnifiedInstaller/install.log
Installing Plone 5.2.3 at /Users/kimadmin/Plone

Creating Python virtual environment.
Installing Python requirements in virtual environment.
Copying Plone-docs
Create buildout: standalone
Creating directory '/Users/kimadmin/Plone/buildout-cache/eggs/cp38'.
Getting distribution for 'buildout.sanitycheck==1.0.2'.
WARNING: The easy_install command is deprecated and will be removed in a future version.
error: Setup script exited with error: SandboxViolation: mkdir('/Users/kimadmin/Library/Caches/com.apple.python/private/var/folders/lx/6chcd57j7w16ycjwnn_x9xg00000gx/T/easy_install-vzq7syn_', 511) {}

The package setup script has attempted to modify files on your system
that are not within the EasyInstall build area, and has been aborted.

This package cannot be safely installed by EasyInstall, and may not
support alternate installation locations even if you run its setup
script by hand.  Please inform the package's author and the EasyInstall
maintainers to find out if a fix or workaround is available.

An error occurred when trying to install /Users/kimadmin/Plone/buildout-cache/downloads/dist/buildout.sanitycheck-1.0.2.zip. Look above this message for any errors that were output by easy_install.
While:
  Installing.
  Loading extensions.
  Getting distribution for 'buildout.sanitycheck==1.0.2'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/kimadmin/Plone/zinstance/lib/python3.8/site-packages/zc/buildout/buildout.py", line 2174, in main
    getattr(buildout, command)(args)
  File "/Users/kimadmin/Plone/zinstance/lib/python3.8/site-packages/zc/buildout/buildout.py", line 658, in install
    self._load_extensions()
  File "/Users/kimadmin/Plone/zinstance/lib/python3.8/site-packages/zc/buildout/buildout.py", line 1182, in _load_extensions
    zc.buildout.easy_install.install(
  File "/Users/kimadmin/Plone/zinstance/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 957, in install
    return installer.install(specs, working_set)
  File "/Users/kimadmin/Plone/zinstance/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 682, in install
    for dist in self._get_dist(requirement, ws):
  File "/Users/kimadmin/Plone/zinstance/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 574, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/Users/kimadmin/Plone/zinstance/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 1745, in _move_to_eggs_dir_and_compile
    [tmp_loc] = glob.glob(os.path.join(tmp_dest, '*'))
ValueError: not enough values to unpack (expected 1, got 0)
Buildout failed. Unable to continue

Installation has failed.
See the detailed installation log at /Users/kimadmin/Downloads/Plone-5.2.3-UnifiedInstaller/install.log
to determine the cause.

Here is the install.log:

Detailed installation log
Starting at Sat Dec 12 15:02:44 CST 2020
Creating Python virtual environment.
Virtualenv successfully created
Installing Python requirements in virtual environment.
Ignoring pywin32: markers 'platform_system == "Windows"' don't match your environment
Ignoring ntfsutils: markers 'platform_system == "Windows" and python_version < "3.0"' don't match your environment
Collecting setuptools==42.0.2 (from -r /Users/kimadmin/Plone/tmp/base_skeleton/requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/54/28/c45d8b54c1339f9644b87663945e54a8503cfef59cf0f65b3ff5dd17cf64/setuptools-42.0.2-py2.py3-none-any.whl
Collecting zc.buildout==2.13.3 (from -r /Users/kimadmin/Plone/tmp/base_skeleton/requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/23/63/feb8f7a73c8974f4764f5bb75a6c06079e962f107db3f7050586638cea80/zc.buildout-2.13.3-py2.py3-none-any.whl
Collecting wheel (from -r /Users/kimadmin/Plone/tmp/base_skeleton/requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/c9/0b/e0fd299d93cd9331657f415085a4956422959897b333e3791dde40bd711d/wheel-0.36.1-py2.py3-none-any.whl
Installing collected packages: setuptools, zc.buildout, wheel
  Found existing installation: setuptools 41.2.0
    Uninstalling setuptools-41.2.0:
      Successfully uninstalled setuptools-41.2.0
Successfully installed setuptools-42.0.2 wheel-0.36.1 zc.buildout-2.13.3
WARNING: You are using pip version 19.2.3, however version 20.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Create buildout: standalone
Copying buildout skeleton
Building Zope/Plone; this takes a while...
Buildout returned an error code: 1; Aborting.
Buildout failed. Unable to continue

To find which pip was used:

$ which pip3
/usr/bin/pip3
nguyenrivers:Plone-5.2.3-UnifiedInstaller kimadmin$ pip3 --version
pip 19.2.3 from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip (python 3.8)

Tried to update pip:

$ pip3 install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: 'RECORD'
Consider using the `--user` option or check the permissions.

WARNING: You are using pip version 19.2.3, however version 20.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
nguyenrivers:Plone-5.2.3-UnifiedInstaller kimadmin$ sudo !!
sudo pip3 install --upgrade pip
Password:
WARNING: The directory '/Users/kimadmin/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/Users/kimadmin/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pip
  Downloading https://files.pythonhosted.org/packages/ab/11/2dc62c5263d9eb322f2f028f7b56cd9d096bb8988fcf82d65fa2e4057afe/pip-20.3.1-py2.py3-none-any.whl (1.5MB)
     |████████████████████████████████| 1.5MB 4.3MB/s 
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
  Rolling back uninstall of pip
  Moving to /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip-19.2.3.dist-info/
   from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/~ip-19.2.3.dist-info
  Moving to /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip/
   from /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/~ip
  Moving to /private/var/root/Library/Caches/com.apple.python/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip/
   from /private/var/root/Library/Caches/com.apple.python/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/~ip
ERROR: Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/usr/local/bin/pip3.8'

WARNING: You are using pip version 19.2.3, however version 20.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Checking if I can try with Python 3.7:

$ brew install python@3.7
[lots o stuff]
Warning: python@3.7 3.7.9_2 is already installed and up-to-date
To reinstall 3.7.9_2, run `brew reinstall python@3.7`
$ brew info python@3.7
python@3.7: stable 3.7.9 (bottled) [keg-only]
Interpreted, interactive, object-oriented programming language
https://www.python.org/
/usr/local/Cellar/python@3.7/3.7.9_2 (4,205 files, 64.5MB)
  Poured from bottle on 2020-11-23 at 09:31:17
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/python@3.7.rb
License: Python-2.0
==> Dependencies
Build: pkg-config ✔
Required: gdbm ✔, openssl@1.1 ✘, readline ✘, sqlite ✘, xz ✔
==> Caveats
Python has been installed as
  /usr/local/opt/python@3.7/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python@3.7/libexec/bin

You can install Python packages with
  /usr/local/opt/python@3.7/bin/pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python

python@3.7 is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.

If you need to have python@3.7 first in your PATH run:
  echo 'export PATH="/usr/local/opt/python@3.7/bin:$PATH"' >> ~/.profile

For compilers to find python@3.7 you may need to set:
  export LDFLAGS="-L/usr/local/opt/python@3.7/lib"

==> Analytics
install: 30,051 (30 days), 73,175 (90 days), 130,840 (365 days)
install-on-request: 24,051 (30 days), 55,215 (90 days), 83,336 (365 days)
build-error: 0 (30 days)
tkimnguyen commented 3 years ago

It worked this time, macOS 10.15.7 Catalina, Python 3.7.9. The problem was the Python. With brew and its Python early in the PATH, the installer runs great. 

$ export PATH="/usr/local/opt/python@3.7/bin:$PATH" 
$ which python3
/usr/local/opt/python@3.7/bin/python3
$ python3
Python 3.7.9 (default, Nov 20 2020, 18:45:38) 
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
$ rm -rf ~/Plone
$ ./install.sh --with-python=`which python3` --password=admin standalone

    Plone 5.2.2 (5209)
    CMF 2.4.8
    Zope 4.5.1
    Python 3.7.9 (default, Nov 20 2020, 18:45:38) [Clang 12.0.0 (clang-1200.0.32.27)]
    PIL 6.2.2 (Pillow)
    WSGI: On
    Server: waitress 1.4.4
tkimnguyen commented 3 years ago

On macOS 10.15.7 Catalina it works with Python 3.8 too:

  $ brew info python@3.8
  $ export PATH="/usr/local/opt/python@3.8/bin:$PATH"
  $ python3 --version
  Python 3.8.6
  $ ./install.sh --with-python=`which python3` --password=admin standalone

    Plone 5.2.2 (5209)
    CMF 2.4.8
    Zope 4.5.1
    Python 3.8.6 (default, Nov 20 2020, 18:29:40) [Clang 12.0.0 (clang-1200.0.32.27)]
    PIL 6.2.2 (Pillow)
    WSGI: On
    Server: waitress 1.4.4
tkimnguyen commented 3 years ago

Works on Ubuntu 20.04 LTS:

    Plone 5.2.2 (5209)
    CMF 2.4.8
    Zope 4.5.1
    Python 3.8.5 (default, Jul 28 2020, 12:59:40) [GCC 9.3.0]
    PIL 6.2.2 (Pillow)
    WSGI: On
    Server: waitress 1.4.4
tkimnguyen commented 3 years ago

Trying out on Windows. Not sure yet what the buildout error is:

\Users\capyb\Plone\python.exe\Scripts\python.exe C:\Users\capyb\Downloads\Plone-5.2.3-Unified-Installer\Plone-5.2.3-UnifiedInstaller\helper_scripts\create_instance.py --uidir=C:\Users\capyb\Downloads\Plone-5.2.3-Unified-Installer\Plone-5.2.3-UnifiedInstaller --plone_home=\Users\capyb\Plone --instance_home=\Users\capyb\Plone\zinstance --itype=standalone --force_build_from_cache=no --password="admin" 
Copying buildout skeleton
Building Zope/Plone; this takes a while...
Parts of the install are missing. Buildout must have failed. Aborting.

C:\Users\capyb\Plone\zinstance>\Users\capyb\Plone\python.exe\Scripts\buildout.exe
Unused options for buildout: 'backups-dir' 'deprecation-warnings'.
Updating instance.
Updating repozo.
Updating zopepy.
Generated interpreter 'C:\\Users\\capyb\\Plone\\zinstance\\bin\\zopepy'.
Updating unifiedinstaller.
Versions had to be automatically picked.
The following part definition lists the versions picked:
[versions]
nt-svcutils = 2.13.0
c:\users\capyb\plone\python.exe\lib\site-packages\zc\buildout\pep425tags.py:77: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
c:\users\capyb\plone\python.exe\lib\site-packages\zc\buildout\pep425tags.py:81: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
  warn=(impl == 'cp')):
c:\users\capyb\plone\python.exe\lib\site-packages\zc\buildout\pep425tags.py:87: RuntimeWarning: Config variable 'Py_UNICODE_SIZE' is unset, Python ABI tag may be incorrect
  sys.version_info < (3, 3))) \
Failed Windows build with error code: 1; Aborting.

After it runs buildout, it thinks some folders are missing but they don't seem to be when I check manually:

 \Users\capyb\Plone\zinstance\bin\buildout
Parts of the install are missing. Buildout must have failed. Aborting.

The code in question is create_instance.py,

    if opt.itype == 'standalone':
        if not (os.path.exists(os.path.join(opt.instance_home, 'bin', 'instance' + ext)) and
                os.path.exists(os.path.join(opt.instance_home, 'parts', 'instance')) and
                os.path.exists(os.path.join(opt.instance_home, 'var'))):
            _print("Parts of the install are missing. Buildout must have failed. Aborting.")
            sys.exit(1)
jensens commented 3 years ago

Maybe nt_svcutils need to get pip installed before buildout. I can have look later the coming week.

tkimnguyen commented 3 years ago

Thanks for the hint! I'll try that. No worries, you've done a lot!!!

tkimnguyen commented 3 years ago

nt_svcutils wasn't the problem. Something else is going on: the instance.exe isn't being created (that is the one test that fails). Going to see if I can figure out why, because no error or warning is now generated by buildout.

tkimnguyen commented 3 years ago

Aha, as per https://pypi.org/project/plone.recipe.zope2instance/#the-generated-control-script:

On the windows platform the bin/instance script as described below will not be generated, because it uses a Unix specific implementation.

To run Plone start it with:

.\bin\runwsgi.exe -v .\parts\etc\wsgi.ini

Or for development in debug mode use:

.\bin\runwsgi.exe -v .\parts\etc\wsgi.ini
tkimnguyen commented 3 years ago

I'm not finding a runwsgi.exe so will try to figure that out next. I see that may have been addressed in this forum thread

fredvd commented 3 years ago

@jensens The 3.7 macos failure works for me locally on Catalina, but it only did on the second try. I had first unpacked the archive in tmp and ran the tests there, but then I got a sandbox violation on the install.sh of the zeoserver & 2 clients. When I ran the tests in a subfolder of my homedir everything runs fine. 

Trying:
    stdout, stderr, returncode = doCommand('%s/zeocluster/bin/plonectl stop' % testTarget)
Expecting nothing
ok
Trying:
    returncode
Expecting:
    0
ok
Trying:
    safestr(stderr)
Expecting:
    ''
ok
1 items passed all tests:
  61 tests in tests-install-unix.rst
61 tests in 1 items.
61 passed and 0 failed.
Test passed.
Done.
(installer)

@ale-rt You mentioned something during the standup about alternative code for checking if ports are open? Now this is used:

def checkport(server="127.0.0.1", port=8080):
    a_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    location = (server, port)
    result_of_check = a_socket.connect_ex(location)
    a_socket.close()
    return result_of_check == 0

As the tests run for me locally on a 'real' Catalina Mac OS X my guess is something is limited/problematic in the virtual environment test runner. (https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.15-Readme.md)

ale-rt commented 3 years ago

@jensens The 3.7 macos failure works for me locally on Catalina, but it only did on the second try. I had first unpacked the archive in tmp and ran the tests there, but then I got a sandbox violation on the install.sh of the zeoserver & 2 clients. When I ran the tests in a subfolder of my homedir everything runs fine.

Might it be a timing issue?

@ale-rt You mentioned something during the standup about alternative code for checking if ports are open? Now this is used:

def checkport(server="127.0.0.1", port=8080):
    a_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    location = (server, port)
    result_of_check = a_socket.connect_ex(location)
    a_socket.close()
    return result_of_check == 0

It was:

import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    if s.connect_ex(("127.0.0.1", 11211)) != 0:
        ...

Which is basically the same that is done here.

As the tests run for me locally on a 'real' Catalina Mac OS X my guess is something is limited/problematic in the virtual environment test runner. (https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.15-Readme.md)

Maybe your system is faster than the GHA executor and the test succeeds for that reason?

ale-rt commented 3 years ago

I checked the error and I see there is already a sleep/retry thing

fredvd commented 3 years ago

Maybe your system is faster than the GHA executor and the test succeeds for that reason?

Lets re-run all jobs and see if it was a random fluke. Python 2.7/3.6 and 3.8 did finish successull on GHA.

You can't rerun individual jobs unless you split all jobs in separate yaml's and not use matrix I understand from this thread :-/ Still very new to GHA. https://github.community/t/re-run-jobs/16145/9