Feature Request: More Docker support

[jlongman]

Is your feature request related to a problem? Please describe.

I'm always frustrated when the flavour doesn't have a linter. Developers I work with are frustrated when they have to pull the all image or have a flavour miss a file-type.

Describe the solution you'd like

Support docker for all linters. Using docker is supported for a swift linter but could be universal. (It's not clear to me whether this is docker in docker or using the docker socket).

For docker-supported situations you would always have a linter. A naive implementation would have an image for each linter - providing the overhead was less than wasted download for unused linters in a flavour this would be a net win. If there is no image for a specific linter an existing flavour can always be used.

Further development could match the required linters with known flavours and do a best set match dynamically. This suggests we could generate more flavour permutations and achieve better fits.

In my experiments I just exposed the docker socket in the megalinter container.

Describe alternatives you've considered

Use the all flavour and download 2+GB to bitbucket and codebuild every time. CodeBuild caching is unpredictable; bitbucket is unpredictable caching large images.

Use a manually selected flavour and error when an unresolved linter happens.

Additional context

I've prototyped it in my fork here: https://github.com/jlongman/mega-linter/commits/docker-socket. You can see changes in json and python linter descriptors as well as the Linter code. The changes are pretty minimal.

The code first checks to see if the linter is in the current image because that is most efficient. Not done, but a sensible thing would be to check for the socket before initializing all linters ie to default use_docker to False if it's not available. Because I was integrating third party containers I added the ability to omit the command name as the image I chose already had a well configured entrypoint but resulted in an invocation of e.g. black black --ignore errors.

I manually build a bare image which has no linters in it.

to give you an idea, this is how I'm running it locally:

docker run  --rm -it \
   -v /var/run/docker.sock:/var/run/docker.sock \
   -v $(pwd):/tmp/lint  \
   -e MEGALINTER_VOLUME_ROOT=$(pwd)   \
   -e LOG_LEVEL \
jlongman/mega-linter:v4.34.0

This isn't directly mergable because my fork still has the python portion built into a separate image which it uses as a base to aid quick local development; it also has (partial, buggy) bitbucket support. It shouldn't materially change the content, but was different enough I didn't just cherry pick it onto your master and create a PR without validating interest first.

This doesn't address docker security. (!)

[nvuillam]

Do you mean something like building Mega-Linter content "on the fly", depending on which file types are found ?

Aren't you afraid that it would run 7 different docker images, and that it would have worse runtime perfs ? :/

Swiftlint is docker based because I didn't find a way to install it on the docker image... and even like that i'm not sure it's really working (there has been some issues about it)

What are the flavors that your devs are missing ? It's quite easy to add new ones, but it has impact on CI , so if you can provide additional flavors to add (and that would match many projects), i'll be glad to add them :)

But in case I didn't get well enough your exxplanation, please could you make a draft PR so I can see all the files updated ?

i'd also love to have native bitbucket support in the yeoman installer located in mega-linter-runner :)

[jlongman]

I had a long draft written and then a reboot took it away. In any case I've noted some other problems with docker so I'm still looking at them.

super short version:

• either precompute high likelihood images assuming that they will be more prevalent OR compose the images on the fly, which implies nothing overlaps - which might be the challenge.
• definitely size is a concern, but there is still value in large installs which ALWAYS validate.
• I can see some subtleties with docker as I noted above, so I'm not surprised with swiftlint
• It's more I'm thinking of a generalized solution than flavors (partially motivated because I disabled your actions in my fork but if it works this would be a better solution for us).
• Honestly I haven't looked at the installer yet, there are still bugs in the bitbucket reporting so I've been focusing on them.

cheers!

[Kurt-von-Laven]

Here is a completely different approach that has been brewing in the back of my mind for others' consideration. Please let me know whether this would address the issue your team is facing, @jlongman, and any considerations I may have overlooked. This is a long-term proposal and entails a great deal of work, so I don't want to engender false hopes that this would be a quick change. What if we created a virtual environment (that could be shared between projects when running Mega-Linter locally) if one wasn't already present, installed all of the necessary tools into it using asdf, and downloaded precisely the linters needed for each project to a centralized cache on-the-fly? This will probably imply writing some new asdf plugins, but asdf already supports an impressive array of tools. It would also imply that we would no longer require users to install Docker to run Mega-Linter locally and no longer need to build or maintain a fleet of Docker images.

In the meantime, are you already aware of FAIL_IF_MISSING_LINTER_IN_FLAVOR, which you set to true if you don't want to have to worry that your flavor may not cover enough linters?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[Kurt-von-Laven]

asdf would also work great within Docker if it's important to keep Docker involved.