Allow "non-repository" input to the analyzer

[sschuberth]

As ORT takes transparency and reproducibility of results serious, currently only local directories that are under version control can serve as the input to the analyzer. This is because for such "working trees" it is machine-readable where they originate from, and also the state of the source code is encoded into the VCS revision. If a remote VCS repository is to be analyzed, it either needs to be manually cloned with the respective VCS tool or the ORT Downloader first.

However, there are use-cases where the source code to analyze never was checked into version control, and committing it to a "fake" repository just to be able to analyze it defeats the purpose of capturing genuine provenance information.

Thus the proposal is to relax / extend the valid input types for the analyzer to the following:

• local directories under version control (currently the only supported input)
• local directories not under version control (or an unsupported VCS; maybe use e.g. the SPDX package verification code as part of provenance information)
• local ZIP files (use e.g. the SHA1 / SHA256 of the archive as part of provenance information)
• Docker images (mount the image as a file system and analyze a directory inside it; use e.g. the Docker image tag / SHA1 as part of provenance information)
• Remote ZIP files / Docker images referred to by URLs (the URL plus hashes could serve as provenance information)

Open questions:

• How to ensure that following ORT tools, that may run on a different machine, has access to the source code of the project as well?
  • For local directories, it could simply be a requirement that all ORT tools have to run on the same machine (or have access to the same file system with the project source code).

Related issues / PRs:

• https://github.com/oss-review-toolkit/ort/issues/2896
• https://github.com/oss-review-toolkit/ort/pull/8764

[fviernau]

What we have missed to discuss so far, but I believe is important to clarify is: Project.vcs / Project.vcsProcessed. Previously, one could always set it. The feature implemented here should probably ensure we can still always set that property. So, its datatype would need to change?!

[sschuberth]

Good point about a Project's provenance info. If the analysis is not performed on a repository, it also does not make sense to store VCS-related information for the Projects, and these should probably be migrated to KnownProvenance as well.

This relates a bit to @mnonnenmacher's questions about how follow-up ORT tools, that might run on a different machine / node, should get access to the provenance's / project's source code.

[pepper-jk]

Would you like me to incorporate those changes to Project into #8764 as well before we proceed?

[sschuberth]

Would you like me to incorporate those changes to Project into #8764 as well before we proceed?

I'm not sure. Your PR is already quite involving, so maybe we should not make it yet more complex, but roll out the changes in stages, if possible.

[pepper-jk]

Yes, I see the problem: either we have breaking changes after every small pull request, or we have a large pull request, which is impossible to review.

Maybe we could create a feature branch for this topic and merge any PRs onto there first and only once it is finished you merge it into main? This way there would only be breaking changes in one release and you also had small PRs to review.

FYI: I have taken a look at the changes required for Project and they appear to be very similar to what I already did in Repository, so I believe adding those on top would be straight forward. It would just takes a bit more work.