Config file: Support Wildcards in directory

blakemorgan commented 5 years ago

Our repo doesn't have a pom.xml file at / but it does in multiple subdirectories (so /*/). Dependabot didn't recognize the wildcard in the directory name when I put that in the config file. Is it possible to get wildcards recognized in the directory value of the config file?

greysteil commented 5 years ago

Frustratingly, this is a bit more difficult on our backend than it sounds. The answer here is pretty similar to:

https://github.com/dependabot/dependabot-core/issues/2177

We'd like to support this, and it's on our roadmap (together with grouping projects), but it's a way off.

bsideup commented 4 years ago

Would be nice to have it!

Here is an example where it fails: https://github.com/bsideup/liiklus/blob/09d16a867b93ca966d6200c6d62f646d0344065b/settings.gradle#L11

alexf101 commented 3 years ago

I wrote a script that auto-generates Dependabot configs using globs 😅

Please feel free to use my awful hacks however you like until Dependabot/Github implement this feature properly.

https://gist.github.com/alexf101/b65cbfe7c5a61df7d925589a71d200cf

jauderho commented 3 years ago

+1 from me.

I have a dockerfiles repo with multiple directories representing different images being built. I was dismayed to find out that dependabot.yml only allows specifying one directory.

Even if we are somehow able to do the following would be a start

directory: | /dir1 /dir2

michaelmerrill commented 3 years ago

Any update on where this is on the roadmap?

blakemorgan commented 2 years ago

For our teams, the Terraform support with Dependabot greatly increased the need for this feature. We have local modules storing common infrastructure for the environments, then a separate folder for each environment. Sometimes, because of weird dependencies, we have to have two folders for each environment. That's upwards of 10 folders for each project. It would be nicer to have one entry in the config file instead of 10.

gjermundgaraba commented 2 years ago

If someone would like a javascript hack as a workaround:

handlebar template for dependabot file

"fast-glob" all the different types you need and generate a new dependabot file:

fb.sync([
    `${repoRoot}/**/package.json`,
    `${repoRoot}/**/go.mod`,
    `${repoRoot}/**/pom.xml`,
    `${repoRoot}/**/Dockerfile`,
    // Stuff to ignore:
    `!${repoRoot}/**/node_modules`,
    `!${repoRoot}/**/vendor`])
    .map(mapItToAStructureYouWantToSendToHandleBar)

For monorepos, it is death trying to maintain the dependabot file by hand.

tmatilai commented 2 years ago

Generating the configuration for each and every Terraform module in a repository worked fine, until:

The property '#/updates' had more items than the allowed 200

No idea how to work around this now. 🙁

highb commented 2 years ago

@greysteil Any estimates on when this may be implemented in Dependabot? Managing our Monorepo with one definition for every requirements.txt and package.json is not sustainable for us.

greysteil commented 2 years ago

Afraid not - I don't work on Dependabot directly anymore and haven't for the last couple of years. Cc @exvuma who can hopefully help more!

exvuma commented 2 years ago

It's not on our roadmap right now tbh. It's a large engineering undertaking but is top of mind on how to address this sort of configurability issue for monorepos. I am tracking this internally. As we nail down our planning I will update this ticket. In the meantime, you can see the features we commit to shipping on our public roadmap.

DennisDyallo commented 2 years ago

Really want this feature. We have multiple csproj files in subdirectories and now we are forced to list them all by path in the dependabot.yml, instead of just doing /rootDir/**.csproj

gaby commented 2 years ago

👍🏻 to get this feature added

prapurnakumariTR commented 1 year ago

Is this feature available?

gtirloni commented 1 year ago

It's not on our roadmap right now tbh. It's a large engineering undertaking but is top of mind on how to address this sort of configurability issue for monorepos. I am tracking this internally. As we nail down our planning I will update this ticket. In the meantime, you can see the features we commit to shipping on our public roadmap.

It's not even in the Future column for dependabot. Is there any interest in adding this feature in 2022?

cyberveseli commented 1 year ago

Vote for this feature 👍

rutalreja-deloitte commented 1 year ago

For TF files, can be modified for any extension A little late to the party, but finally here

One Liner that I run in a git actions pipeline (remember to give pipeline push permissions) from root of repo if [[ "$(find . -name '*.tf' -not -path '*.terraform*' -printf '%h\n' | sort -u | wc -l)" -lt "200" ]]; then yq -i 'del(.updates.[])' .github/dependabot.yml && for DIR in $(find . -name '*.tf' -not -path '*.terraform*' -printf '%h\n' | sort -u); do dir=${DIR:1} yq -i '."updates"+=[{"package-ecosystem":"terraform" | . style="double","directory":env(dir) | . style="double","schedule":{"interval":"daily" | . style="double"}}]' .github/dependabot.yml; done;fi

A little explaination:

Firstly it uses yq, if you cant deal with the dependency, enjoy doing it with jq yourself
find . -name '*.tf' -not -path '*.terraform*' -printf '%h\n' | sort -u finds the names of the folders that contain .tf files apart from inside the .terraform folder
The first if condition if [[ ... -lt "200" ]] checks you dont have more than 200 folders, as mentioned above. If you do good luck.
yq -i 'del(.updates.[])' .github/dependabot.yml this clears your dependabot updates yaml array to remake it in the next line.
- If you have other things you dont want to delete you will have to figure out what is old and what is new if you are running in a pipeline, a head start for that would be something like diff <(command) <(command) to compare
Last for DIR in $(find . -name '*.tf' -not -path '*.terraform*' -printf '%h\n' | sort -u); do dir=${DIR:1} yq -i '."updates"+=[{"package-ecosystem":"terraform" | . style="double","directory":env(dir) | . style="double","schedule":{"interval":"daily" | . style="double"}}]' .github/dependabot.yml loops through, creates the structure and fills in the directory

Snailedlt commented 1 year ago

Would be neat if directory could take a list or an array of paths too... maybe something like this?

  - package-ecosystem: "composer"
    directory: 
      - "/app"
      - "/"
    schedule:
      interval: "daily"

fullstackzach commented 1 year ago

I would like to throw my hat in the ring to request this feature too. We use a monorepo that has several package.json files in different folders in the project.

All I am looking to do is create the dependabot.yml in order to add a prefix to the commit message, but that means I would have to list all of the locations of package.json files explicitly.

Interestingly without a dependabot.yml, dependabot security updates appears to be able to automatically detect where these package.json files exist. It would actually be nice if directory was optional and it followed the auto-detect behavior if it was not included.

Makeshift commented 1 year ago

Since this has been going on for a while and probably isn't going to be added soon, I created a little workaround in the form of an action that generates dependabot.yml based on globs.

Example template -> Example output

https://github.com/marketplace/actions/generate-dependabot-glob

It's exceptionally simple and was created in a rush as opposed to with good practice, but hopefully is enough for simple projects/monorepos or can be used as a base to build your own generator from. It's certainly a lot nicer to have a 20 line file that generates into a 300 line one automatically!

Dreamsorcerer commented 1 year ago

Thanks to #6255, I've just found a partial workaround for Python projects.

You can add a top-level requirements file which includes all your other requirements files (with -r sub/requirements.txt). It will then check them all recursively. Example: https://github.com/aio-libs/aiohttp-demos/pull/540/files

Obviously doesn't resolve the discoverability issue (you still need to hardcode every requirements file), but it means you'll now get 1 PR to update a dependency across all requirements files, instead of a separate PR for each file.

gaby commented 11 months ago

4 years later, are there any updates on this?

kashifest commented 11 months ago

This is really desirable now, we have for example four different go modules in sub directories in https://github.com/metal3-io/cluster-api-provider-metal3/ , we would really love to put the directories in an array and possibly also hope the updates of one dependancy goes into one PR no matter if it is updating multiple gomodules or not.

arslanbekov commented 10 months ago

After added the ability to groups feature, there is very needed support for specifying wildcard in the directory.

Dreamsorcerer commented 9 months ago

Thanks to #6255, I've just found a partial workaround for Python projects.

It appears this no longer works. I've just had 8 PRs open for the same dependency. And every PR fails because the versions obviously conflict with each other.

Dreamsorcerer commented 9 months ago

Thanks to #6255, I've just found a partial workaround for Python projects.

It appears this no longer works. I've just had 8 PRs open for the same dependency. And every PR fails because the versions obviously conflict with each other.

I don't know if someone saw that message and put in a fix or something, but seems to be batching again today.

Dreamsorcerer commented 9 months ago

Oh, I think I just realised it was caused by a security update. So, security updates came in one at a time, which all failed, then the regular version update did the normal batch update, and made the other PRs redundant.

dgokcin commented 8 months ago

has anyone tried the described options in this github blog post? looks promising.

ilmax commented 8 months ago

@dgokcin I am using it and it somehow works, what I can't do with grouped updates is to ignore a dependency version that's part of the group. The command is there in the dependabot comment but it looks like is not doing anything.

RDhar commented 8 months ago

@dgokcin I am using it and it somehow works, what I can't do with grouped updates is to ignore a dependency version that's part of the group. The command is there in the dependabot comment but it looks like is not doing anything.

Mind sharing a link to an example where you've somehow got wildcard-directory-support working within dependabot.yml?

dgokcin commented 8 months ago

@dgokcin I am using it and it somehow works, what I can't do with grouped updates is to ignore a dependency version that's part of the group. The command is there in the dependabot comment but it looks like is not doing anything.

@ilmax thank you for sharing what has worked and what has not worked for you. as @rdhar said an example would be great.

ilmax commented 8 months ago

@dgokcin I am using it and it somehow works, what I can't do with grouped updates is to ignore a dependency version that's part of the group. The command is there in the dependabot comment but it looks like is not doing anything.

Mind sharing a link to an example where you've somehow got wildcard-directory-support working within dependabot.yml?

Wait, I may have misunderstood a bit. the linked post is about grouped updates and that's what I was referring to in my comment. This issue on the contrary is about having dependabot support some sort of search within the project folders. This second part is still not working, but if you want an example of the first one, I can post it here

cprice404 commented 8 months ago

This has turned into a blocker for us for a few of our repos. The repos have 10+ subdirectories that have example code in them using different combinations of dependencies, and listing each one as a package-ecosystem means that we regularly have a dozen dependabot PRs open that are all consuming Actions runners. It is looking like we will have to disable dependabot because that many PRs/actions are not sustainable for us, and we expect the number of directories to grow.

devbeard commented 6 months ago

Just want to add a nuget perspective. I know - preaching to the choir.

Typical large dotnet projects have multiple project files (aka manifests for nugets). The projects can optionally be linked through a solution file, which is just a grouping construct really, or a "workspace" file. The solution typically define an app, and the projects define the app's components.

When configuring dependabot with the newest grouping feature, you can have multiple dependencies updated in a single PR per project, but there is no way of updating one dependency across projects, which would be ideal.

Example:

A major library like Newtonsoft is updated
In our solution, 15 out of 30 projects use this dependency (random numbers, but not unthinkable)
Ideal outcome: One PR to update all projects to the same version
Currently: 15 PRs and 15 sequential commits, where each of the commits leave main in a stage of "consolidation error" until the last one where all versions across the solution are again in sync.

My current workaround is to create a new PR, updating the deps in a single commit, and just ignoring the PRs from the bot.

jan-xyz commented 6 months ago

You could also change the base branch of the PRs to one of the 15 other PRs and merge them all together before merging to main.

cpswan commented 6 months ago

To do what @jan-xyz suggests I have a little script - rollup.sh

More explanation in the accompanying blog post Roll up, rollup, get your Dependabot PRs together here

ascopes commented 4 months ago

Any update on this? I have a project for a Maven plugin that has integration tests taking the shape of multiple maven projects in src/it/*/pom.xml that would benefit from this feature.

reggermont commented 4 months ago

Any update on this? I have a project for a Maven plugin that has integration tests taking the shape of multiple maven projects in src/it/*/pom.xml that would benefit from this feature.

The issue is 5 years old, pretty sure you can forget about it

burik666 commented 4 months ago

we need it

gioce90 commented 4 months ago

We absolutely need this feature. Anything yet?

Maybe replacing the directory field with a directories:

updates:
  - package-ecosystem: "nuget"
    directories:
    - "/App/First" 
    - "/App/Second"
    open-pull-requests-limit: 15
    # other settings...

Dreamsorcerer commented 4 months ago

Possibly the same status as https://github.com/dependabot/dependabot-core/issues/4364#issuecomment-1887660591

PrimeDominus commented 3 months ago

Woah! How is this not a standard feature already???

jonasfroeller commented 3 months ago

This issue is from 2019. How is this not implemented yet?

adamlui commented 3 months ago

I'd like it too...

Parth-1612 commented 3 months ago

same

Mayeu commented 3 months ago

If any of you really need this feature then vote with your feet and migrate to another dependency manager, like Renovate (https://github.com/renovatebot/renovate) it's almost 0 config with autodetection of any number of files needed to be updated.

(I'm not affiliated with this tool, I'm just a happy user since I took the 15 minutes needed to migrate.)

gioce90 commented 3 months ago

If any of you really need this feature then vote with your feet and migrate to another dependency manager, like Renovate (https://github.com/renovatebot/renovate) it's almost 0 config with autodetection of any number of files needed to be updated.

(I'm not affiliated with this tool, I'm just a happy user since I took the 15 minutes needed to migrate.)

I came to the same conclusion a few weeks ago (I'm not affiliated too). Renovate is a really great alternative, especially outside of GitHub (I was looking for something that worked for Azure DevOps).

Snailedlt commented 3 months ago

If any of you really need this feature then vote with your feet and migrate to another dependency manager, like Renovate (https://github.com/renovatebot/renovate) it's almost 0 config with autodetection of any number of files needed to be updated.

(I'm not affiliated with this tool, I'm just a happy user since I took the 15 minutes needed to migrate.)

I came to the same conclusion a few weeks ago (I'm not affiliated too). Renovate is a really great alternative, especially outside of GitHub (I was looking for something that worked for Azure DevOps).

Agreed, though I personally prefer dependabot due to its tight integration on GitHub.

Just FYI, dependabot also works on Azure DevOps :)

robinshin commented 2 months ago

Same issue here

abdulapopoola commented 2 months ago

The team has started working on this and hoping we have some releases soon. Tagging @jurre , @jakecoffman , and @Nishnha

dependabot / dependabot-core

Config file: Support Wildcards in directory #2178