Project organising

[grigi]

Hi, I've recently discovered this project and I really like it's focus and simplicity. Apologies if this description is a little too wordy.

This is an initial project organising PR.

What I've done here is:

• Create a Makefile as a way to manage the project from the command line, currently only manages test dependency pinning, linting & testing. I find doing make up, make format, make ci etc... quite convenient.
• make up will try to do a clean-install of dependencies.
• Moved mypi.ini into setup.cfg and also handle test/coverage settings in that file
• Setup coverage on the unittest runs.
• With bump to mypy 1.5 mypy issue 6462 is not affecting this anymore, and removed the comments around that.
• Update: redis==5.0.0 is not working perfectly.

I was considering changing format.sh and run_checks.sh, but decided against it as I want your feedback first.

Anything I did here is up for change.

Rambling

This is purely out of selfish reasons where I need a simple and reliable queue manager that can run asyncio code. All the established ones are sync (Celery/rq), or in a weird limbo (arq). This seems to fit my requirements quite well.

It would need a few more things such as :

• Ability to expose the Monitoring front-end as a FastAPI page
• Documentation (I was thinking readthedocs.io, Sphinx (as you have relatively thorough docstrings) and piccolo-theme
• Clarity on how to setup a task to be closer-to-exactly-once or at-least-once-with-idempotency (e.g. for better reliability, if you can declare a task as idempotent, you can er on the chance of accidentally running multiple times, but if not idempotent you really shouldn't)

Less important, but useful:

• Ability to generate prometheus stats
• Tracing (we use Datadog's ddtrace, but that is supposedly OpenTracing compatible) so you can see where the task came from

I'll likely be working on a few of the above points in any case.

[tobymao]

@grigi thanks for this contribution, really appreciate it, left a couple comments, happy to discuss

[grigi]

Hi, I removed green, used coverage directly instead.

The reason I committed tests/requirements.txt was so that we won't get unexpected test failures during development, but only when on explicitly updates the test dependencies. Since you didn't want it I removed it and changed make up to simulate a clean install and update packages eagerly. This way you should be able to keep the dev environment a little cleaner.

Turns out I was wrong, and redis==5.0.0 does break something. I backed off on allowing it to upgrade for now.

[grigi]

There is a few janitorial things I would like to discuss:

1. Move the [dev] dependencies into tests/requirements.txt because it's not ideal to allow users to install library dev dependencies.
2. Clean up the remaining shell scripts in the root, in particular format.sh and run_checks.sh as they can be replaced with relevant Makefile commands.