search

conda-incubator / conda-store

Data science environments, for collaboration. ✨
https://conda.store
BSD 3-Clause "New" or "Revised" License
150 stars 50 forks source link

[CI] Verbose flag `-vv` cannot be passed as pytest parameter to the current tests #849

Closed peytondmurray closed 3 months ago

peytondmurray commented 4 months ago

Context

Currently celery workers get passed any command line args you pass to pytest, making it impossible to see verbose output. This issue is to track work required to change this.

Value and/or benefit

We can't debug effectively unless we can get verbose output.

Anything else?

No response

peytondmurray commented 4 months ago

I added the following lines to my python 3.10's argparse.py:2174:

import traceback
traceback.print_stack()

When running pytest tests/test_app_api.py::test_conda_store_register_environment_workflow -svv I get the following traceback from the affected celery worker:

File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/contrib/testing/worker.py", line 73, in start
    return super().start()
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/worker/worker.py", line 202, in start
    self.blueprint.start(self)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 340, in start
    blueprint.start(self)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 746, in start
    c.loop(*c.loop_args())
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/worker/loops.py", line 130, in synloop
    connection.drain_events(timeout=2.0)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/connection.py", line 341, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 997, in drain_events
    get(self._deliver, timeout=timeout)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/utils/scheduling.py", line 55, in get
    return self.fun(resource, callback, **kwargs)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 1035, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 754, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 414, in _poll
    return cycle.get(callback)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/utils/scheduling.py", line 55, in get
    return self.fun(resource, callback, **kwargs)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 418, in _get_and_deliver
    callback(message, queue)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 1017, in _deliver
    callback(message)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/transport/virtual/base.py", line 639, in _callback
    return callback(message)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/kombu/messaging.py", line 656, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/worker/consumer/consumer.py", line 685, in on_task_received
    strategy(
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/worker/strategy.py", line 207, in task_message_handler
    handle(req)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/worker/worker.py", line 225, in _process_task
    req.execute_using_pool(self.pool)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/worker/request.py", line 754, in execute_using_pool
    result = apply_async(
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/concurrency/base.py", line 153, in apply_async
    return self.on_apply(target, args, kwargs,
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/concurrency/base.py", line 30, in apply_target
    ret = target(*args, **kwargs)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/app/trace.py", line 630, in trace_task_ret
    R, I, T, Rstr = trace_task(app.tasks[name],
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/app/trace.py", line 586, in trace_task
    return task.__trace__(uuid, args, kwargs, request)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/celery/app/trace.py", line 453, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/pdmurray/Desktop/workspace/conda-store/conda-store-server/conda_store_server/_internal/worker/tasks.py", line 104, in task_update_storage_metrics
    conda_store = self.worker.conda_store
  File "/home/pdmurray/Desktop/workspace/conda-store/conda-store-server/conda_store_server/_internal/worker/tasks.py", line 51, in worker
    self._worker.initialize()
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
  File "/home/pdmurray/Desktop/workspace/conda-store/conda-store-server/conda_store_server/_internal/worker/app.py", line 63, in initialize
    super().initialize(*args, **kwargs)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/traitlets/config/application.py", line 465, in initialize
    self.parse_command_line(argv)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/traitlets/config/application.py", line 881, in parse_command_line
    self.cli_config = deepcopy(loader.load_config())
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/traitlets/config/loader.py", line 895, in load_config
    self._parse_args(argv)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/traitlets/config/loader.py", line 954, in _parse_args
    self.parsed_data = self.parser.parse_args(to_parse)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/argparse.py", line 1833, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/site-packages/traitlets/config/loader.py", line 790, in parse_known_args
    return super().parse_known_args(args, namespace)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/argparse.py", line 1866, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/argparse.py", line 2079, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/argparse.py", line 2009, in consume_optional
    arg_count = match_argument(action, selected_patterns)
  File "/home/pdmurray/.pyenv/versions/miniconda3-latest/envs/conda-store-server-dev/lib/python3.10/argparse.py", line 2175, in _match_argument
    traceback.print_stack()
peytondmurray commented 4 months ago

I don't think we're passing these arguments on purpose to the celery task, but traitlets allows for configuration variables to be set from the command line, and it automatically parses command line args. I think we can explicitly pass an empty argv to Application.initialize to prevent this.