New connector structure and single source of truth for configuration

[SYNchroACK]

Problems

• connectors settings are not organized and are not easy to maintain
• docker related files require a lot of unnecessary customization

Example

Using MISP connector as example, regarding

• settings - mentions to settings can be found in different formats in the following files:
  • external-import/misp/README.md
  • external-import/misp/docker-compose.yml
  • external-import/misp/src/config.yml.sample
• docker - can be generic and more portable:
  • Dockerfile (WORKDIR, ENTRYPOINT)
  • docker-compose.yaml (env_file)

Suggestions

• Settings should always passed via environment variables, yaml should be deprecated. File external-import/misp/src/config.yml.sample should be removed.
• Instead of having .yml file, the connector should provide a .env.dist file, which then will be easily copied to .env file with the settings defined by the user. This .env file would be also automatically used by the docker-compose.yml file with the service definition of the connector, which again, removes the need to also have the configuration parameters hard-coded in the docker-compose.yml (yet another file to keep up to date).
• For developers who need to test/debug, they only need to execute set -a ; source .env and then python misp.py and everything will work. Also note that .env.dist file always has a comment (prefixed by #) where it starts with [Required] or [Optional], followed by the description of the config parameter.
• requirements.txt should be at same directory level of src/, not inside of src/.
• connector python "entrypoint" should always have the same name src/connector.py.
• entrypoint.sh is not necessary, should be removed and replaced with a generic ENTRYPOINT entry in Dockerfile
• Dockerfile can have WORKDIR defined and can be generic like /opt/connector/.

Commit Example

I show exactly what I'm proposing on the following commit: https://github.com/SYNchroACK/opencti-connectors/commit/c7ae154dd30ccb980594889fe31a4fa37ad1fc95

[SYNchroACK]

@SamuelHassine if you agree, I could help on this work.

[SamuelHassine]

Hello @SYNchroACK,

We've already an on-going work https://github.com/OpenCTI-Platform/client-python/pull/288 for rewriting the whole connector base class with integrated test suit and more guided development guidelines.

The work has been started by @nor3th and we're taking over right now.

Kind regards, Samuel

[SYNchroACK]

Ok cool. Let me know if you need feedback.