search

BCDA-APS / bluesky_training

Bluesky training, including instrument package
https://bcda-aps.github.io/bluesky_training/
Other
11 stars 0 forks source link

code from session_logs now in apstools.utils #83

Closed prjemian closed 1 year ago

prjemian commented 1 year ago
prjemian commented 1 year ago

@rodolakis : Can you review these changes? The code here and in apstools needs other work to enable the choice of a constant log directory. (The fix we applied today won't work when the functions are called from apstools.utils.) I indicated what the fix will need to be with a TODO marking.

prjemian commented 1 year ago

@rodolakis - Don't bother with the review yet. There is a problem with the upstream package requiring a fix in apstools:

        if maxBytes > 0 or backupCount > 0:
>           handler = logging.handlers.RotatingFileHandler(log_file, maxBytes=maxBytes, backupCount=backupCount)
E           AttributeError: module 'logging' has no attribute 'handlers'. Did you mean: '_handlers'?

../../../micromamba-root/envs/anaconda-test-env-py-3.9/lib/python3.10/site-packages/apstools/utils/log_utils.py:64: AttributeError

Not sure I understand why Python 3.9 tests would be trying to run with Python 3.10 installation packages. Something wrong in the CI here at at least.

Marking this PR as draft for now.

prjemian commented 1 year ago

Curious, most recent CI change is because Python 3.9 process was installing Python 3.10:

2023-03-23T21:21:32.5883378Z   python                         3.10.9        he550d4f_0_cpython       conda-forge
prjemian commented 1 year ago

Cannot test with Py3.11 yet:

error    libmamba Could not solve for environment specs
      The following packages are incompatible
      ├─ python 3.11**  is installable with the potential options
      │  ├─ python 3.11.0, which can be installed;
      │  └─ python 3.11.0 would require
      │     └─ python_abi 3.11.* *_cp311, which can be installed;
      └─ tiled   is uninstallable because it requires
         ├─ tiled-formats [0.1.0a65 hd8ed1ab_0|0.1.0a65 hd8ed1ab_1|...|0.1.0a75 hd8ed1ab_0], which requires
         │  └─ tiled-base [>=0.1.0a.65,<0.1.1a |>=0.1.0a.66,<0.1.1a |...|>=0.1.0a.75,<0.1.1a ] but there are no viable options
         │     ├─ tiled-base [0.1.0a77|0.1.0a79|...|0.1.0a[89](https://github.com/BCDA-APS/bluesky_training/actions/runs/4505458935/jobs/7931203294#step:3:91)] would require
         │     │  └─ sparse >=0.13.0 , which requires
         │     │     └─ numba >=0.49  but there are no viable options
         │     │        ├─ numba [0.49.0|0.49.1|...|0.53.1] would require
         │     │        │  ├─ python >=3.6,<3.7.0a0 , which conflicts with any installable versions previously reported;
         │     │        │  └─ python_abi 3.6.* *_cp36m, which conflicts with any installable versions previously reported;
         │     │        ├─ numba [0.49.0|0.49.1|...|0.56.3] would require
         │     │        │  ├─ python >=3.7,<3.8.0a0 , which conflicts with any installable versions previously reported;
         │     │        │  └─ python_abi 3.7.* *_cp37m, which conflicts with any installable versions previously reported;
         │     │        ├─ numba [0.49.0|0.49.1|...|0.56.4] would require
         │     │        │  ├─ python >=3.8,<3.9.0a0 , which conflicts with any installable versions previously reported;
         │     │        │  └─ python_abi 3.8.* *_cp38, which conflicts with any installable versions previously reported;
         │     │        ├─ numba [0.53.0|0.53.1|...|0.56.4] would require
         │     │        │  ├─ python >=3.9,<3.10.0a0 , which conflicts with any installable versions previously reported;
         │     │        │  └─ python_abi 3.9.* *_cp39, which conflicts with any installable versions previously reported;
         │     │        ├─ numba [0.55.0|0.55.1|...|0.56.4] would require
         │     │        │  ├─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
         │     │        │  └─ python_abi 3.10.* *_cp310, which conflicts with any installable versions previously reported;
         │     │        ├─ numba [0.49.0|0.49.1|...|0.53.1] would require
         │     │        │  └─ python >=3.6,<3.7.0a0 , which conflicts with any installable versions previously reported;
         │     │        ├─ numba [0.49.0|0.49.1|...|0.56.4] would require
         │     │        │  └─ python >=3.7,<3.8.0a0 , which conflicts with any installable versions previously reported;
         │     │        ├─ numba [0.49.0|0.49.1|...|0.56.4] would require
         │     │        │  └─ python >=3.8,<3.9.0a0 , which conflicts with any installable versions previously reported;
         │     │        ├─ numba [0.53.0|0.53.1|...|0.56.4] would require
         │     │        │  └─ python >=3.9,<3.10.0a0 , which conflicts with any installable versions previously reported;
         │     │        └─ numba [0.55.0|0.55.1|0.56.3|0.56.4] would require
         │     │           └─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
         │     ├─ tiled-base [0.1.0a65|0.1.0a66|...|0.1.0a75] would require
         │     │  ├─ python >=3.10,<3.11.0a0 , which conflicts with any installable versions previously reported;
         │     │  └─ python_abi 3.10.* *_cp310, which conflicts with any installable versions previously reported;
         │     ├─ tiled-base [0.1.0a65|0.1.0a66|...|0.1.0a75] would require
         │     │  ├─ python >=3.7,<3.8.0a0 , which conflicts with any installable versions previously reported;
         │     │  └─ python_abi 3.7.* *_cp37m, which conflicts with any installable versions previously reported;
         │     ├─ tiled-base [0.1.0a65|0.1.0a66|...|0.1.0a75] would require
         │     │  ├─ python >=3.8,<3.9.0a0 , which conflicts with any installable versions previously reported;
         │     │  └─ python_abi 3.8.* *_cp38, which conflicts with any installable versions previously reported;
         │     └─ tiled-base [0.1.0a65|0.1.0a66|...|0.1.0a75] would require
         │        ├─ python >=3.9,<3.10.0a0 , which conflicts with any installable versions previously reported;
         │        └─ python_abi 3.9.* *_cp39, which conflicts with any installable versions previously reported;
         └─ tiled-server [0.1.0a77 hd8ed1ab_0|0.1.0a79 hd8ed1ab_0|...|0.1.0a89 hd8ed1ab_0], which requires
            └─ tiled-base [0.1.0a77 pyhd8ed1ab_0|0.1.0a79 pyhd8ed1ab_0|...|0.1.0a89 pyhd8ed1ab_0], which cannot be installed (as previously explained).
  critical libmamba Could not solve for environment specs
  Error: Failed to execute ["/home/runner/micromamba-bin/micromamba create -n anaconda-test-env-py-3.11 -y --log-level warning \"coveralls\" \"pytest\" \"pytest-cov\" \"python=3.11\" \"setuptools-scm\" -f /home/runner/work/bluesky_training/bluesky_training/environment.yml"]: Error: The process '/home/runner/micromamba-bin/micromamba' failed with exit code 1
  /home/runner/work/_actions/mamba-org/provision-with-micromamba/main/dist/main/index.js:666[93](https://github.com/BCDA-APS/bluesky_training/actions/runs/4505458935/jobs/7931203294#step:3:95)
      throw Error(`Failed to execute ${JSON.stringify(args)}: ${error}`)
prjemian commented 1 year ago

That's tiled. See if it can be pip installed instead.

prjemian commented 1 year ago

Also: https://github.com/BCDA-APS/bluesky_training/blob/a43023c609f8e62573f99c4854eb8bda410c5b30/.github/workflows/unit-tests-pytest.yml#L117

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

prjemian commented 1 year ago

Now the error is with the queueserver:

``` 2023-03-23T22:02:37.8212896Z 2023-03-23T22:02:37.8213635Z EnvironmentNameNotFound: Could not find conda environment: training_2022 2023-03-23T22:02:37.8214784Z You can list all discoverable environments with `conda info --envs`. 2023-03-23T22:02:37.8215051Z 2023-03-23T22:02:37.8215136Z 2023-03-23T22:02:38.5289080Z Traceback (most recent call last): 2023-03-23T22:02:38.5290098Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/bin/start-re-manager", line 6, in 2023-03-23T22:02:38.5290765Z from bluesky_queueserver.manager.start_manager import start_manager 2023-03-23T22:02:38.5291666Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/bluesky_queueserver/manager/start_manager.py", line 8, in 2023-03-23T22:02:38.5292298Z from .manager import RunEngineManager 2023-03-23T22:02:38.5295929Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/bluesky_queueserver/manager/manager.py", line 21, in 2023-03-23T22:02:38.5296468Z from .plan_queue_ops import PlanQueueOperations 2023-03-23T22:02:38.5297200Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/bluesky_queueserver/manager/plan_queue_ops.py", line 1, in 2023-03-23T22:02:38.5297668Z import aioredis 2023-03-23T22:02:38.5298274Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/aioredis/__init__.py", line 1, in 2023-03-23T22:02:38.5298754Z from aioredis.client import Redis, StrictRedis 2023-03-23T22:02:38.5299403Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/aioredis/client.py", line 32, in 2023-03-23T22:02:38.5299868Z from aioredis.connection import ( 2023-03-23T22:02:38.5300903Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/aioredis/connection.py", line 33, in 2023-03-23T22:02:38.5301356Z from .exceptions import ( 2023-03-23T22:02:38.5301977Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/aioredis/exceptions.py", line 14, in 2023-03-23T22:02:38.5302544Z class TimeoutError(asyncio.TimeoutError, builtins.TimeoutError, RedisError): 2023-03-23T22:02:38.5302954Z TypeError: duplicate base class TimeoutError 2023-03-23T22:02:47.5121072Z bluesky_queueserver-training is not running 2023-03-23T22:02:47.8858985Z Traceback (most recent call last): 2023-03-23T22:02:47.8860392Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/bin/qserver", line 6, in 2023-03-23T22:02:47.8861476Z from bluesky_queueserver.manager.qserver_cli import qserver 2023-03-23T22:02:47.8864530Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/bluesky_queueserver/manager/qserver_cli.py", line 20, in 2023-03-23T22:02:47.9051634Z from .plan_queue_ops import PlanQueueOperations 2023-03-23T22:02:47.9053225Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/bluesky_queueserver/manager/plan_queue_ops.py", line 1, in 2023-03-23T22:02:47.9053971Z import aioredis 2023-03-23T22:02:47.9055353Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/aioredis/__init__.py", line 1, in 2023-03-23T22:02:47.9056121Z from aioredis.client import Redis, StrictRedis 2023-03-23T22:02:47.9057025Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/aioredis/client.py", line 32, in 2023-03-23T22:02:47.9107480Z from aioredis.connection import ( 2023-03-23T22:02:47.9108192Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/aioredis/connection.py", line 33, in 2023-03-23T22:02:47.9108664Z from .exceptions import ( 2023-03-23T22:02:47.9109275Z File "/home/runner/micromamba-root/envs/anaconda-test-env-py-3.11/lib/python3.11/site-packages/aioredis/exceptions.py", line 14, in 2023-03-23T22:02:47.9109837Z class TimeoutError(asyncio.TimeoutError, builtins.TimeoutError, RedisError): 2023-03-23T22:02:47.9110250Z TypeError: duplicate base class TimeoutError ```

Might be related to the 0.0.18 release. Try dropping back to previous 0.0.17.

prjemian commented 1 year ago

Py3.11 needs queueserver 0.0.18, so remove testing on Py3.11 for now.

prjemian commented 1 year ago

Again, this error:

        if maxBytes > 0 or backupCount > 0:
>           handler = logging.handlers.RotatingFileHandler(log_file, maxBytes=maxBytes, backupCount=backupCount)
E           AttributeError: module 'logging' has no attribute 'handlers'

../../../micromamba-root/envs/anaconda-test-env-py-3.9/lib/python3.9/site-packages/apstools/utils/log_utils.py:64: AttributeError

Yet the Python 3.9 documentation says that logging.handlers.RotatingFileHandler() is available. So this might be an import problem (needing import logging.handlers in apstools upstream?)?

prjemian commented 1 year ago

Same problem found with Python 3.8.

prjemian commented 1 year ago

Must apply fixes in apstools and make a new release there before continuing here. Ugh.

prjemian commented 1 year ago

New apstools release 1.6.15 is available now. Re-run the CI.

prjemian commented 1 year ago

First re-run did not get the newest apstools. Wait a few hours, then re-run.

coveralls commented 1 year ago

Pull Request Test Coverage Report for Build 4513828882

Changes Missing Coverage Covered Lines Changed/Added Lines %
bluesky/instrument/session_logs.py 10 11 90.91%
<!-- Total: 10 11 90.91% -->
Totals Coverage Status
Change from base Build 4504812792: -0.2%
Covered Lines: 822
Relevant Lines: 1028
💛 - Coveralls
tguruswamy commented 1 year ago

Tested basic functionality and looks good.

$ ipython3
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:08:06) [GCC 11.3.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.11.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from instrument.collection import *
/home/beams2/TGURUSWAMY/src/python/bluesky/instrument/_iconfig.py
Activating auto-logging. Current session state plus future input saved.
Filename       : /home/beams2/TGURUSWAMY/src/python/bluesky/.logs/ipython_console.log
...
$ vim instrument/iconfig.yml 
$ ipython3
Python 3.10.10 | packaged by conda-forge | (main, Mar 24 2023, 20:08:06) [GCC 11.3.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.11.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from instrument.collection import *
/home/beams2/TGURUSWAMY/src/python/bluesky/instrument/_iconfig.py
Activating auto-logging. Current session state plus future input saved.
Filename       : /tmp/ipython_console.log
...