search

pydiverse / pydiverse.pipedag

A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
https://pydiversepipedag.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Switch from poetry (pypi) to pixi (conda-forge) #207

Closed windiana42 closed 1 week ago

windiana42 commented 2 months ago

Checklist

NMAC427 commented 1 month ago

I'm not sure if you're aware of this, but currently the CI tests don't run properly. All the calls to pytest return withing a few seconds, and only a fraction of the selected tests are actually being executed.

windiana42 commented 1 month ago

I'm not sure if you're aware of this, but currently the CI tests don't run properly. All the calls to pytest return withing a few seconds, and only a fraction of the selected tests are actually being executed.

@NMAC427 good catch! We did find a considerable amount of problems after switching to pixi and even figured out that snowflake tests never really ran on the main branch.

The problem vanishes when removing the --workers flag. We had to do something about logger initialization since the 'no:terminal' plugin did not prevent serialization problems any more (this deteriorated debug output since quite some time):

    # register dummy terminal reporter since it is needed by pytest even with
    # plugins:"no:terminal" option
    terminal_reporter = TerminalReporter(config, DontPrint())
    config.pluginmanager.register(terminal_reporter, "terminalreporter")

Commenting out these 4 lines does increase runtime of running all tests with --workers flag again.