Refactor CTFCLI to more OOP approach

[MilyMilo]

So, this is a complete refactor of ctfcli as discussed. Notable changes:

Architecture

• Modules - ctfcli has been separated into two main modules, it's not only a CLI but also sort of an SDK. The core module now packages logic previously found inside utils, wrapped into classes.
• Exceptions - The classes in the core module will only print out warnings, that do not interrupt the process. Everything else will be thrown as an exception, so that they can be caught and handled however desired inside both the CLI, and other use-cases / implementations.
• Validation and Error Handling - I tried to set-up as much validation as possible. The cli should now print-out nice and useful error messages if something goes wrong, instead of full stack-traces because some property in the challenge.yml file hasn't been defined.
• Typing - Both cli and core have type hints everywhere.

Functionality

• I've added a try/except clause around the CLI entrypoint to catch an exception when project is not initialised. It will now ask the user if he/she would like to initialise a new project in the current directory.
• I've removed the finalize method as it was not useful. In the future we might replace it with something that would perhaps create a skeleton challenge.yml file, that would act as a starting point for filling it. It might pre-fill some fields that it knows, e.g the challenge name based on the directory name.
• deploy can now deploy ALL deployable challenges, and that's the default behaviour if the challenge parameter is not provided.
  • For the SSH and Registry deployments, to facilitate this behaviour the challenge name will be automatically appended. So from now on the host should be for example: registry://registry.example.com/example-project and the challenge name will be appended for a full location.
• deploy will now also automatically login to the registry with Cloud and Registry deployments.
  • For the cloud deployment - it requires the instance url to be ctfd assigned (example.ctfd.io) - this is because to log-in to the registry we require a username like admin@example.ctfd.io and resolving that canonical domain name could be a hassle and introduce issues. The deployment will use the access token as the password.
  • For the registry deployment - it will look for the username and password keys inside a [registry] section in the project config file.
• I've added a --hidden switch to ctf challenge install which will deploy the challenge / challenges in a hidden state regardless of their challenge.yml value.
• I've added ctf challenge edit <name> and ctf challenge edit <name> --dockerfile to open challenge.yml or Dockerfile for that challenge - which I found useful during testing and I think is really useful overall.
• I've added an aliases for dir (path) and for view (show) which i think will be useful.
• DEBUG logging has been added, for the most part it will just print executed commands. To facilitate this ctfcli will read a LOGLEVEL environment variable.
• I've added progress bars for deploy / install / sync

Testing

• Unit tests have been added for the entire core module, the coverage is at 97%.
• Testing for the CLI is still TODO.

Other

• ctfcli will only work on python 3.10+ now, because I've used some typing definitions as well as match statements.
• I've moved to using poetry instead of the compile-in-docker approach. pyproject.toml is a standardised approach to handling package definitions, which I think we should use. pep-518
• Everything uses pathlib.Path now - previously this was mixed with os.join, and pathlib is much more pleasant to work with, and generally superior.
  • With that, most whatever_dir variables have been renamed to whavever_path - to be clear that it will return a pathlib path.
• The CLI will now exit with an exit code 0 if everything went right, and 1 if something went wrong. This might be useful in pipelines. With install / sync / deploy - exit code will be 1 if ANY of the challenges failed to install / sync / deploy.
• typing_extensions package has been added to support some of typing not available in python 3.10.
• python-fire has been updated to 0.5.0 - because it used to print out the output of executed commands. However if we want to use it to assign the exit-code it had to be suppressed which was only possible with 0.5.0.
• pytest has been added as a test runner with pytest-sugar and pytest-cov.

Breaking Changes:

• Pretty much everything. Most plugins will be broken - even though the plugin interface has not been changed.

---

Remaining TODOs:

• [ ] Add some usage documentation
• [ ] Check the build and publish workflow, as I did not adjust it
• [ ] Check how poetry can replace setup.py with poetry build
• [ ] Perhaps move away from using Makefile and package scripts inside the pyproject.toml file: poetry scripts
• [ ] Check how ctfcli can be installed with tools like pipx - which is an approach I think we should document and recommend.

When I was refactoring the code I've came-up with several other ideas for features which I'll create separate issues for.

[pl4nty]

Looking forward to seeing this merge, it'll simplify some CI/CD pipelines I'm hoping to release.

If you're open to changes, I had one request - could ctf install automatically perform a ctf sync if a challenge has a duplicate name? Currently I have to try ctf sync for each chal, then run ctf install if sync fails

[MilyMilo]

@pl4nty Sure it's possible and an easy change, but I'm just wondering if a desirable one.

While it's better from a CI/CD point of view where you can be sure you're just installing a challenge which might already be installed. However, we're mostly using this as a CLI - in which case it's most likely just a human error (either wrong command, or a wrong challenge - and we can't know for sure). We could end-up accidentally syncing challenges that are unfinished.

I think I'd see this as either a switch or a new command. e.g

ctf challenge install-or-sync or ctf challenge install --sync-if-exists

CC: @ColdHeat what do you think?

[pl4nty]

Thanks, a switch or new command would work well for my usecase. I'm currently grep-ing for the sync error to detect if install is required, which is pretty brittle (breaks on this PR)

[ColdHeat]

I like the idea of having install also do a sync if asked to do so. I think there's some other questions about where is the source of truth but I think a switch is enough. What about --force?

[MilyMilo]

@ColdHeat @pl4nty I actually think that doing a sync instead of installing the challenge as a duplicate is a better behaviour for --force. I'll adjust that.

[pl4nty]

Installing from git doesn't seem to be fixed, ie pip install git+https://github.com/CTFd/ctfcli. It was also failing before the branch was merged. Maybe this snipped is needed in pyproject.toml?

[tool.poetry.scripts]
ctf = "ctfcli.__main__:main"