Create multiple outputs for Cylc Flow package

kinow commented 3 years ago

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[ ] Re-rendered with the latest conda-smithy (Use the phrase code>@<space/conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

Closes #6

First commit adds missing dependencies from cylc-flow master branch (aiofiles, pyuv). Second commit splits the recipe into multiple outputs. Namely:

cylc-flow-base, similar to jupyterhub-base, and matplotlib-base; this -base appears to be the common pattern for base conda recipes
cylc-flow-empty, imports cylc-flow-base, adds empy
cylc-flow-report-timings, imports cylc-flow-base, adds pandas
cylc-flow-main_loop, imports cylc-flow-base, adds pympler and matplotlib-base (this is not pip/SetupTools package, but the Conda matplotlib-base that doesn't include GUI libs like Qt)

The last item is a combination of all the requirements in the SetupTools dependency groups main_loop-log_data_store, main_loop-log_main_loop, and main_loop-log_memory. If others prefer, we can have one output for each dependency group in pip, but I thought it would be easier to maintain the conda recipe this way?

WIP as this would trigger a new release. Created PR because I wanted a break from the UI to reset my brain :nerd_face: and since this would be necessary anyway for the next release (or at least before the final 8 release)

conda-forge-linter commented 3 years ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the 'cylc-flow-empy' output doesn't have any tests.
It looks like the 'cylc-flow-report-timings' output doesn't have any tests.
It looks like the 'cylc-flow-main_loop' output doesn't have any tests.
License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found here.

conda-forge-linter commented 3 years ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found here.

conda-forge-linter commented 3 years ago

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

kinow commented 3 years ago

Pending:

[x] before the release, bump up the version and build numbers
[x] check if we have any other dependencies missing
[ ] thorough tests locally building the package to confirm the generated packages work as expected (both individually in an empty conda env, as well as with cylc-uiserver, cylc-ui, etc)
[ ] update installation instructions

kinow commented 3 years ago

I had forgotten about this PR :disappointed_relieved: Will try again before the next release.

hjoliver commented 3 years ago

So it's not ready to go in for the beta-0 release?

kinow commented 3 years ago

So it's not ready to go in for the beta-0 release?

No, I haven't finished testing it. I wrote how I thought it had to look like, but didn't test locally to compare if the produced conda files were correct, nor had time to test with the other packages and confirm it works. Sorry :disappointed_relieved:

kinow commented 3 years ago

Rebased, and bumped build ID. Let's see what errors we have pending. Will test locally too.

kinow commented 3 years ago

Testing with conda build recipe/

Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed
failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform linux-64: {"ansimarkup[version='>=1.0.0']", "graphene[version='>=2.1,<3']", "jinja2[version='2.11.0,<2.12']", "rx[version='>=1.6,<2']", "urwid[version='>=2.0,<2.1']", "pyuv[version='>=1.4.0,<1.5.0']", "metomi-isodatetime[version='>=1!2.0.2,<1!2.1.0']", "protobuf[version='>=3.15.0,<3.16.0']"}
WARNING:conda_build.build:failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform linux-64: {"ansimarkup[version='>=1.0.0']", "graphene[version='>=2.1,<3']", "jinja2[version='2.11.0,<2.12']", "rx[version='>=1.6,<2']", "urwid[version='>=2.0,<2.1']", "pyuv[version='>=1.4.0,<1.5.0']", "metomi-isodatetime[version='>=1!2.0.2,<1!2.1.0']", "protobuf[version='>=3.15.0,<3.16.0']"}
WARNING conda_build.build:test(2859): failed to get install actions, retrying.  exception was: Unsatisfiable dependencies for platform linux-64: {"ansimarkup[version='>=1.0.0']", "graphene[version='>=2.1,<3']", "jinja2[version='2.11.0,<2.12']", "rx[version='>=1.6,<2']", "urwid[version='>=2.0,<2.1']", "pyuv[version='>=1.4.0,<1.5.0']", "metomi-isodatetime[version='>=1!2.0.2,<1!2.1.0']", "protobuf[version='>=3.15.0,<3.16.0']"}
Tests failed for cylc-flow-base-8.0b0-py39_2.tar.bz2 - moving package to /home/kinow/Development/python/miniconda3/envs/delete/conda-bld/broken
WARNING:conda_build.build:Tests failed for cylc-flow-base-8.0b0-py39_2.tar.bz2 - moving package to /home/kinow/Development/python/miniconda3/envs/delete/conda-bld/broken
WARNING conda_build.build:tests_failed(2955): Tests failed for cylc-flow-base-8.0b0-py39_2.tar.bz2 - moving package to /home/kinow/Development/python/miniconda3/envs/delete/conda-bld/broken
TESTS FAILED: cylc-flow-base-8.0b0-py39_2.tar.bz2

:sighs:

kinow commented 3 years ago

Tested again today in the NIWA notebook. Took ages to build with conda build recipe (which gave me some time to read about another workflow manager and compare with cylc 😬 ).

Worked with no issues this time!

kinow commented 3 years ago

Except I cannot install any of the packages as I'm using Py 3.9 🤣

Let's create a new env with Py 3.8

kinow commented 3 years ago

Installed latest Miniconda, then created an env with py3.7.3. Built successfully.

# To have conda build upload to anaconda.org automatically, use
# conda config --set anaconda_upload yes
anaconda upload \
    /home/kinow/miniconda3/envs/test/conda-bld/linux-64/cylc-flow-base-8.0b0-py37_2.tar.bz2 \
    /home/kinow/miniconda3/envs/test/conda-bld/linux-64/cylc-flow-empy-8.0b0-py37_2.tar.bz2 \
    /home/kinow/miniconda3/envs/test/conda-bld/linux-64/cylc-flow-main_loop-8.0b0-py37_2.tar.bz2 \
    /home/kinow/miniconda3/envs/test/conda-bld/linux-64/cylc-flow-report-timings-8.0b0-py37_2.tar.bz2
anaconda_upload is not set.  Not uploading wheels: []

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "cylc-flow-base-8.0b0-py37_2": {
    "recipe": {}
  },
  "cylc-flow-empy-8.0b0-py37_2": {
    "recipe": {}
  },
  "cylc-flow-main_loop-8.0b0-py37_2": {
    "recipe": {}
  },
  "cylc-flow-report-timings-8.0b0-py37_2": {
    "recipe": {}
  }
}

####################################################################################
Resource usage summary:

Total time: 0:14:33.9
CPU usage: sys=0:00:00.3, user=0:00:01.6
Maximum memory usage observed: 49.2M
Total disk usage observed (not including envs): 2.7K

I only have the old Cylc in my $PATH.

$ cylc --version
7.8.6-7-gf3abe

Then installed cylc-flow conda install -c file:///home/kinow/miniconda3/envs/test/conda-bld/linux-64/ cylc-flow-base. And testing:

(test) kinow@kinow-VirtualBox:~/Development/python/workspace/cylc-flow-feedstock$ cylc --version
8.0b0
(test) kinow@kinow-VirtualBox:~/Development/python/workspace/cylc-flow-feedstock$ cd ~/cylc-suites/five
(test) kinow@kinow-VirtualBox:~/cylc-suites/five$ cylc install
INSTALLED five from /home/kinow/cylc-suites/five -> /home/kinow/cylc-run/five/run6
(test) kinow@kinow-VirtualBox:~/cylc-suites/five$ cylc play --no-detach five/run6
                                                                                    ._.                                                     
                                                                                    | |                                                     
                                                                        ._____._. ._| |_____.                                               
                                                                        | .___| | | | | .___|         The Cylc Workflow Engine [8.0b0]      
                                                                        | !___| !_! | | !___.           Copyright (C) 2008-2021 NIWA        
                                                                        !_____!___. |_!_____!   & British Crown (Met Office) & Contributors.
                                                                              .___! |                                                       
                                                                              !_____!                                                       
2021-04-09T02:02:43Z INFO - Cold Start 20130808T0000Z
2021-04-09T02:02:43Z INFO - Suite server: url=tcp://kinow-VirtualBox:43062/ pid=101707
2021-04-09T02:02:43Z INFO - Suite publisher: url=tcp://kinow-VirtualBox:43038
2021-04-09T02:02:43Z INFO - Run: (re)start=0 log=1
2021-04-09T02:02:43Z INFO - Cylc version: 8.0b0
2021-04-09T02:02:43Z INFO - Run mode: live
2021-04-09T02:02:43Z INFO - Initial point: 20130808T0000Z
2021-04-09T02:02:43Z INFO - Final point: None
2021-04-09T02:02:44Z INFO - [prep.20130808T0000Z] -submit-num=01, host=kinow-VirtualBox
2021-04-09T02:02:44Z INFO - [prep.20130808T0000Z] -triggered off []
^C2021-04-09T02:02:44Z ERROR -
    Traceback (most recent call last):
      File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/runners.py", line 43, in run
        return loop.run_until_complete(main)
      File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
        self.run_forever()
      File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/base_events.py", line 539, in run_forever
        self._run_once()
      File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/base_events.py", line 1739, in _run_once
        event_list = self._selector.select(timeout)
      File "/home/kinow/miniconda3/envs/test/lib/python3.7/selectors.py", line 468, in select
        fd_event_list = self._selector.poll(timeout, max_ev)
    KeyboardInterrupt
    During handling of the above exception, another exception occurred:
    Traceback (most recent call last):
      File "/home/kinow/miniconda3/envs/test/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 580, in start_scheduler
        await self.main_loop()
      File "/home/kinow/miniconda3/envs/test/lib/python3.7/site-packages/cylc/flow/scheduler.py", line 1491, in main_loop
        await asyncio.sleep(duration)
      File "/home/kinow/miniconda3/envs/test/lib/python3.7/asyncio/tasks.py", line 568, in sleep
        return await future
    concurrent.futures._base.CancelledError
2021-04-09T02:02:44Z CRITICAL - Suite shutting down

Also confirmed I cannot import from EmPy.

(test) kinow@kinow-VirtualBox:~/cylc-suites/five$ python -c 'import em'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'em'

Then installed cylc-flow-empy: conda install -c file:///home/kinow/miniconda3/envs/test/conda-bld/linux-64/ cylc-flow-empy

And voilà!

(test) kinow@kinow-VirtualBox:~/cylc-suites/five$ python -c 'import em'
(test) kinow@kinow-VirtualBox:~/cylc-suites/five$

:rocket: :tada:

kinow commented 3 years ago

Just needs to be rebased either now or after next release, and then have @hjoliver or @oliver-sanders to use the generated artefacts and confirm I didn't miss anything. I think the number of outputs hasn't changed, but if we want more we just need to copy an existing output and modify accordingly here :+1:

conda-forge / cylc-flow-feedstock

Create multiple outputs for Cylc Flow package #11