dotnet new gitignore should synch up with public template

asears commented 2 years ago

Summary of issue

The dotnet new gitignore command saves a workflow step when scaffolding projects and really happy to see dotnet new --install Microsoft.DotNet.Common.ItemTemplates::6.0.100 and those workflow improvements.

Drift between gitignore templates or subtractions in the templates might introduce some security issues if secrets are checked in incorrectly for a new project or while doing a project refresh.

As an example, patches from issues updated in a Feb, 2021 template here are not in a June, 2021 template on GitHub.

Repro

This is the first template which comes up in search results for Visual Studio gitignore when scaffolding a new project.
https://github.com/github/gitignore/blob/master/VisualStudio.gitignore

It is different than the template generated using dotnet new gitignore from the 6.0.100 cli. https://github.com/dotnet/templating/blob/main/template_feed/Microsoft.DotNet.Common.ItemTemplates/content/Gitignore/.gitignore

The template here is missing nuget.config comment placeholder. There's specific scenarios where nuget.config should or shouldn't be committed in 2021 and a comment indicating so rather than just removing nuget.config could be shown. See https://github.com/NuGet/Home/issues/10566 and https://github.com/github/gitignore/pull/3706 - I don't know if this situation is resolved but it's impacting in terms of public/private feed credentials. Also don't know if it's responsibility of this repo's maintainers to maintain the other repo's gitignore.
The templating currently excludes .tye files here but not in Github. Would be nice to have a unique identifier (not a url) and some more description about what .tye files are in the .gitignore to look up more info on what tye is and why someone would be ignoring the files from a binary, transitory or security perspective. In any case these should be in synch.

Suggestions

Suggest some background task to synch this repo from GitHub gitignore, as two repos code / PRs diverge and improvements happen.
Source the template from github directly, with some option to download / refresh it for offline consumers.
The template currently provided with dotnet new includes some additional exclusions for mac settings and things that are not really mac settings. Add these to the Github template if required, though are they really needed or is the workflow of a unique developer that could be added to personal/global gitignore? When I see common .gitignore entries for mac, the gitignore list is a lot smaller and usually just include .DS_Store and thumbnails. and config.make, tarballs, etc.
The mac ignores look a little broad (eg. *.tar.gz is not mac-specific, it's a binary archive format) and what is "Mac bundle stuff". The comments could be more specific and professional if introduced via this repo's contributions. These will be in every dotnet project.

As a templating library, could enforce further rules and guidance for .gitignore comments, security and ensure standardization in the gitignore for dotnet.

Files to sync:

File	Source	Target
`.gitignore`	github/gitignore	dotnet/sdk
`template.json`	dotnet/templating	SchemaStore/schemastore - bi-directional
`docs`	dotnet/templating	dotnet/templating/wiki

bekir-ozturk commented 2 years ago

Hi @asears, Thanks for the issue. We will consider automating this in the future, though we don't have a defined timeline for this yet. @baronfel Can you take a look at the suggestions section?

baronfel commented 2 years ago

After talking to @vlada-shubina we have two use cases for this kind of automatic update:

.gitignore files, which should be sourced from the gitignore repo as @asears mentioned, and
.editorconfig files, which VS has a default mechanism for (according to this Docs page), but we should figure out where the backing source is and ingest from that. @sayedihashimi do you know who would be a good contact for me to run that down?

I think two examples is enough for us to look into making some kind of sync task, either a manual task that whoever is on maintenance detail (like the MSBuild kitten rotation) would do, or make some kind of automation around this.

Regarding the specific .gitignore changes proposed by @asears, I'd suggest raising those issues at the gitignore repo, because I do think we should move to just sourcing the data from that repository. Single source of truth and all that.

sayedihashimi commented 2 years ago

I think @kendrahavens may be able to answer the editorconfig question, but not sure.

kendrahavens commented 2 years ago

I'll pass this to @mikadumont @jmarolf :)

jmarolf commented 2 years ago

do you know who would be a good contact for me to run that down?

As Kendra says @baronfel that would be me or Mika.

The editorconfig that we generate when the user runs dotnet new editorconfig lives here and I have been considering how we will keep it up-to-date in releases of the SDK.

I would like some mechanism so that when the user gets notified when some kinds of templates are out of date. At the very least some explicit command to refresh a template item (dotnet renew editorconfig ¯_(ツ)_/¯) would be nice. in the case editorconfig it is unlikely that we would ever have new options available outside of an SDK release so for us a solution could be dotnet migrate being aware of new editorconfig options. Feel free to schedule some time to talk about this if you want: https://aka.ms/jmarolf

cremor commented 2 years ago

in the case editorconfig it is unlikely that we would ever have new options available outside of an SDK release

Maybe in the future, but as of now there are already missing rules in the EditorConfig file, see #4393

agilenut commented 2 years ago

Any update on this? If the sync mechanism is still a ways out, would it be possible to get a few updates made to the editorconfig via PR in the meantime?

baronfel commented 2 years ago

We don't have plans to do the sync mechanism for the .NET 7 timeframe currently, but we are open to periodic syncs. There's one happening over at https://github.com/dotnet/templating/pull/4598 right now, if you'd like to review and ensure it does what you'd expect it to!

vlada-shubina commented 2 years ago

We should also sync template.json schema with schemastore. Schema in templating repo: https://github.com/dotnet/templating/blob/main/src/Microsoft.TemplateEngine.Orchestrator.RunnableProjects/Schemas/JSON/template.json Schema in schemastore: https://github.com/SchemaStore/schemastore/blob/master/src/schemas/json/template.json

JanKrivanek commented 2 years ago

We might want to try investigate following tools:

vlada-shubina commented 2 years ago

It would be also good to sync templating/docs folder with Wiki.

JanKrivanek commented 2 years ago

Maestro bot seems to update sources as well (sample PR) - we might want to have a quick look on its' capabilities as well

JanKrivanek commented 1 year ago

@GangWang01 - please have a look what might be the options for this problem and list some here

cremor commented 1 year ago

As of SDK version 7.0.100 the .editorconfig template is missing the following rules:

dotnet_style_namespace_match_folder
dotnet_style_prefer_foreach_explicit_cast_in_source

csharp_style_implicit_object_creation_when_type_is_apparent
csharp_style_namespace_declarations
csharp_style_prefer_extended_property_pattern
csharp_style_prefer_local_over_anonymous_function
csharp_style_prefer_method_group_conversion
csharp_style_prefer_null_check_over_type_check
csharp_style_prefer_readonly_struct
csharp_style_prefer_top_level_statements
csharp_style_prefer_tuple_swap
csharp_style_prefer_utf8_string_literals

dotnet_style_allow_multiple_blank_lines_experimental
dotnet_style_allow_statement_immediately_after_block_experimental
csharp_style_allow_blank_line_after_colon_in_constructor_initializer_experimental
csharp_style_allow_blank_lines_between_consecutive_braces_experimental
csharp_style_allow_embedded_statements_on_same_line_experimental

GangWang01 commented 1 year ago

@GangWang01 - please have a look what might be the options for this problem and list some here

Repo File Sync Action
This is a comprehensive action that syncs files from source repo to target repo using pull request. It relies on source repo configures the workflow running the action, specifies which files to sync and the target repo where the change is merged into. Once pushing change to specified files, a pull request is created in target repo syncing the change. It requires GitHub token to aceess the repositories.
For source repo and target repo owned by the same user/org, it can directly create PR branch in target repo.
For source repo and target repo owned by different users/orgs, better way is to use its Fork option with a bot/user GitHub account. It forks target repo under the given GitHub account and create PR branch in the fork. Unfortunately there is a blocking issue https://github.com/BetaHuhn/repo-file-sync-action/issues/270 in this way.
Submodule Sync This action is configured in target repo. It checks if git submodule has new commit with a schedule. When git submodule has new change, this action creates a pull request with the change to submodule folder in parent repo. GitHub can list files that have change.
Git submodule action is not a good option since any change including some files we don't want to sync will create the pull request.
Implement a GitHub workflow/action in target repo that checks regularly if source file has change and create pull request to merge the change.

vlada-shubina commented 1 year ago

Possible way for 2nd option: https://git-scm.com/docs/git-filter-branch

vlada-shubina commented 1 year ago

As of SDK version 7.0.100 the .editorconfig template is missing the following rules:

dotnet_style_namespace_match_folder
dotnet_style_prefer_foreach_explicit_cast_in_source

csharp_style_implicit_object_creation_when_type_is_apparent
csharp_style_namespace_declarations
csharp_style_prefer_extended_property_pattern
csharp_style_prefer_local_over_anonymous_function
csharp_style_prefer_method_group_conversion
csharp_style_prefer_null_check_over_type_check
csharp_style_prefer_readonly_struct
csharp_style_prefer_top_level_statements
csharp_style_prefer_tuple_swap
csharp_style_prefer_utf8_string_literals

dotnet_style_allow_multiple_blank_lines_experimental
dotnet_style_allow_statement_immediately_after_block_experimental
csharp_style_allow_blank_line_after_colon_in_constructor_initializer_experimental
csharp_style_allow_blank_lines_between_consecutive_braces_experimental
csharp_style_allow_embedded_statements_on_same_line_experimental

@cremor it is not possible to sync editorconfig anymore, as it is being generated based on number of rules instead of Visual Studio. Please open the separate issue for editorconfig issues and we can discuss it there.

The files that we are looking at in this issue:

File	Source	Target
`.gitignore`	github/gitignore	dotnet/sdk
`template.json`	dotnet/templating	SchemaStore/schemastore - bi-directional
`docs`	dotnet/templating	dotnet/templating/wiki

cremor commented 1 year ago

@vlada-shubina See https://github.com/dotnet/templating/issues/4393#issuecomment-1321941999

dotnet / templating