Support for `.dot` directories for storing plans

thrix commented 2 years ago

Seems in some cases this would be useful to users.

❯ find
.
./.fmf
./.fmf/version
./.foo
./.foo/bar
./.foo/bar/plan.fmf

❯ tmt plan ls
<no output>

❯ tmt
Found 0 tests.
Found 0 plans.
Found 0 stories.

I wonder if there is a technical limitation why this does not work.

FrNecas commented 2 years ago

.dot directories seem to be ignored, as well as files:

https://github.com/teemtee/fmf/blob/main/fmf/base.py#L479 https://github.com/teemtee/fmf/blob/main/fmf/base.py#L502

Looks to be intentional, I am not sure what the reasoning is.

FrNecas commented 2 years ago

Introduced in https://github.com/teemtee/fmf/commit/e38e78bea158029547cb6dae8c6ffb7daafd18e3 (still no reasoning message :/ )

psss commented 2 years ago

Commit e38e78b is one of the first commits of the fmf project, so frankly I don't remember exactly what the motivation was but most probably:

Prevent ugly node names such as /.plans/.hidden/plan
Speed up fmf tree construction by skipping special folders like .git or .cache and similar

In addition to slowdown, traversing hidden folders could possibly cause problems only if there are files with the .fmf extension. Not sure if that would happen with .git or .pycache. I see two options:

Enable searching hidden files and directories by default
Introduce an .fmf/config file where the desired behaviour could be selected

If we choose the latter way, I'd suggest to use yaml for storing the options. So perhaps something like this:

hidden: true

@jscotka, please add your thoughts and concerns mentioned in the email. Thanks.

richm commented 2 years ago

If you allow traversing .dot directories, you'll need some way to exclude certain directories from being traversed e.g. .git/, .venv/, .tox/, and probably at least a dozen more. Alternately, you could have a config setting which is an allow list - only search in these specified directories. I realize part of the feature set of tmt is the automatic detection of all of the .fmf files in the project - but perhaps there should be some way to restrict where tmt searches?

psss commented 2 years ago

If we need more control over what is being explored, the config could look like this:

explore:
    hidden: true
    include:
      - .plans
    exclude:
      - .git
      - .cache

This would give us some nice flexibility.

jscotka commented 2 years ago

Hi, this sounds like good compromise, because traversing of these hidden files could be bad for performance, especially symlinks there could cause troubles, but it is not so much issue of hidden files, rather than using symlinks some symlink inside .git may point to git root dir itself I can imagine, it can cause cycles, and it is not solved inside TMT and very bad behaviour. But in case of ignoring symlinks issues what can multiply the problem to exponential timeee, the root problem is performance and using .dotted directory for some runtime info, so theoretically these directories may be very deep, and contain lots of subdirs, although there are no fmf files

So as part of this issue we should consider how to solve symlinks,

resolve symlink and in case it is out of git tree just ignore and do not continue, question is if someone uses this as feature ;-) just imagine what happen what happenes, when there will be symlink to root dir / and it will try to traverse though /proc or /sys dir
how to deal with cycles -> stop it, it is not feature anyhow.

What leads us back to deeper configuration of TMT, what were discussed long time ago. it is exactly very good example of config how to change behaviour of TMT, and then you can also somehow e.g.

customize command line defaults
change default plan
configure whatever profiles means (it is very close to changing of defaults)
and whatever you can imagne or change What I'm mentioning this is that from one side it is very easy feature, how to change behaviour of tmt for traverse of hidden files, from second perspective we should think, that this additional customization of configuration of tmt and there is more features what are tight to this.

And another idea is, that if someone needs hidden plans, tests or whatever, so why we should not allow to use metadata inside .fmf directory, it is good place, and everything will be hidden under this subdir?

psss commented 2 years ago

some symlink inside .git may point to git root dir itself I can imagine, it can cause cycles, and it is not solved inside TMT and very bad behaviour.

Symlink loops should be already covered by 0f60389.

it is exactly very good example of config how to change behaviour of TMT

I'm not sure that tmt behaviour or its specific features should be affected by fmf configuration as fmf does not know anything about tmt plans or whatever metadata definition.

And another idea is, that if someone needs hidden plans, tests or whatever, so why we should not allow to use metadata inside .fmf directory, it is good place, and everything will be hidden under this subdir?

I believe the .fmf directory should be kept as a special one and no metadata should ever be stored there as its original purpose was to allocate some space for fmf-related config and similar stuff.

jscotka commented 2 years ago

some symlink inside .git may point to git root dir itself I can imagine, it can cause cycles, and it is not solved inside TMT and very bad behaviour.

Symlink loops should be already covered by 0f60389.

Nice, forgot about that, it is already solved

it is exactly very good example of config how to change behaviour of TMT

I'm not sure that tmt behaviour or its specific features should be affected by fmf configuration as fmf does not know anything about tmt plans or whatever metadata definition.

Right, this is big question, as you said, it is FMF specific onfiguration, and my examples were about TMT config. This confused me. So there are two solutions, first is that there will be no configuration on FMF level, but FMF will allow to use another parameters for functions, and TMT will have it in config. Or second way. fmf will allow you to generally store some config into FMF, so that TMT then will be able to use FMF config storage for own purpose. then it could be merged together and this will be hadled by fmf and tmt parts will be hadled by TMT, btw, has it as part of fmf metadata does also make sense e.g. main.fmf in root dir

.config:
    explore:
      hidden: true
      include:
        - .plans
      exclude:
        - .git
        - .cache
     tmt:
       command_line_defaults:
          - some optiobn

And another idea is, that if someone needs hidden plans, tests or whatever, so why we should not allow to use metadata inside .fmf directory, it is good place, and everything will be hidden under this subdir?

I believe the .fmf directory should be kept as a special one and no metadata should ever be stored there as its original purpose was to allocate some space for fmf-related config and similar stuff.

yep, maybe, just quick solution, how to fix and allow to not add more hidden dirs and magic.

kornys commented 1 year ago

+1 for support dot folders

LecrisUT commented 1 year ago

I believe the .fmf directory should be kept as a special one and no metadata should ever be stored there as its original purpose was to allocate some space for fmf-related config and similar stuff.

I'm not sure about this. I would agree that .fmf should not contain project metadata, but in this case these would be instructions on how to read the current fmf-tree. To me it feels more natural to have that there, rather then in a main.fmf. That would make it more flexible for extensions. I do agree though that it should be kept minimal.

This would also decouple such that .fmf files only contain the metadata of the project, and not of fmf library

kornys commented 8 months ago

In strimzi project we use tmt plans and tests for to running our maven/java tests on testing farm using packit. We would like to have all plans/tests and exec script stored in .tmt or .fmf folder to not mess maven repository with tmt folder in packages. It is a usual convention of storing CI stuff in dot folders (.github, .gitlab). Since tmt is for our project just wrapper for running maven tests in testing farm we really would like to have hidden in dot folder.

https://github.com/strimzi/strimzi-kafka-operator

current tmt related files location -> https://github.com/strimzi/strimzi-kafka-operator/tree/main/systemtest/tmt

LecrisUT commented 8 months ago

@kornys Things have evolved quite a bit since this issue was first created, and in principle all of the tools can now specify a path where to run from. This works only if you don't need the full repo tree, otherwise you will need this issue resolved.

Here is one of my example projects spglib, where I am using 2 trees in .distro and test.

kornys commented 8 months ago

@LecrisUT nice solution, thanks for sharing. Anyway in my case we really need full tree of repository for build all packages and container images before run the testscript.

psss commented 8 months ago

@kornys, what about starting with implementing a minimal solution like this?

explore:
    include:
      - .plans

This config file providing additional directories where the metadata should be explored would be stored under .fmf/config.

kornys commented 8 months ago

@psss that would be perfect 🙂

teemtee / fmf

Support for `.dot` directories for storing plans #183