Support redirects

[RyanCavanaugh]

We should allow a folder structure which is

/some-lib
  redirect.json

The contents of redirect.json would be something that tells types-publisher how to enlist and parse the linked repo. We can probably start with just git URLs and expand as needed

[blakeembrey]

What do you think the specificity of this file should be? Per directory or per version or per commit? Directory seems to make sense for the most part. Does the redirect structure just need to follow the @types structure (E.g. versioned sub-folders, I think files must be the same name as the typed name?). Will typings be able to expand over multiple files and specify the main?

[blakeembrey]

Has the structure of DefinitelyTyped been finalised now? I think the best approach to redirects, and one that keeps things consistent, would be to use git submodules. The advantage here is GitHub has a nifty feature that'll redirect when you click on another GitHub submodule. The only issue here is who has onus on updating the submodule references.

[felixfbecker]

+1 for git submodules.

Currently we have a lot of high-quality typings in the registry that are well-maintained, have their own tests, issue tracker, PRs, docs, and CI. It is easy to contribute to them (no need to pull down all the other thousand typings), and the typings structure follow the library structure (instead of 10,000+ LOC in one file). At the same time, the typings for these exact libraries are still maintained in DT, even though often outdated compared to types/. People are basically duplicating their efforts, because they might not even know that an equivalent in types/ exists. We really need some way to redirect them from DT, and I think submodules would be an excellent solution, because of the mentioned GitHub feature. They are generally frowned upon because the submodules don't get recursively cloned by default. But for DT, this would be exactly what we want - if you clone DT, you want to modify some typings in there. If you want to modify types/ typings, you can clone that repo directly.

We currently use subfolders for different versions of typings and could easily give them an individual package.json with a version field and proper @types/ name.

[felixfbecker]

@mhegazy https://github.com/Microsoft/types-publisher/issues/52#issuecomment-243186214

We would be happy to take PRs for this feature if anyone is interested in contributing. most of the infrastructure is already in place.

It would be nice if one of the TypeScript members could comment on the submodules proposal before putting work into a PR :)

[mhegazy]

The open question for submodules, as noted by @blakeembrey, is the update strategy. is the publisher responsible for always updating to latest? or is the package author responsible for this whenever they are ready to push a new version out? if it is the publisher how often should it look for these package changes? currently we have a webhook that is listening on definitelytyped, and ideally we would like keep it this way.

[felixfbecker]

Currently, when we change on of the repos, we update the commit hash manually in the typings registry. That can be done really quickly with the GitHub online editor, but there were some ideas floating around for having a bot that does this or a typings publish command. For updating the git submodule, the typings maintainer should be responsible, just as he is for updating the registry entry. I guess you cannot do it online though, so it needs to happen with a PR. Updating happens afaik through git submodule update --remote when inside the submodule dir, but I dont know a lot about submodules. It may be worth to find out if this can be done programmatically through the GitHub API so typings publish can automatically open a PR at DT. @blakeembrey what do you think?

[blakeembrey]

I'm not really sure, there's advantages on both sides and I believe I have presented each side (though offline, not particularly here). Hand editing is definitely made easier with text files, but it's possible we can make a bot to update submodules and submit PRs also. I imagine we'll want to do that flow so that tests can be run on the PR and an external change doesn't break DefinitelyTyped. Since both methods can be automated, sticking to the more valuable one (likely submodules, since the GitHub interface is friendly towards it) would be better.

[felixfbecker]

@RyanCavanaugh So what is going on with this? Quoting yourself from https://github.com/Microsoft/TypeScript/issues/9184#issuecomment-226303784:

For large and popular libraries, I agree that having a separate repo is the way to go. We fully intend to support that scenario by allowing redirection from DefinitelyTyped to other repos so that well-maintained libraries can have their own repo.

We have 350 typings in the registry in total and of those are 200 typings in the @types org ranging from the types of node itself to the most-used libraries on NPM like express. These are well-maintained by a core team and can be improved with PRs aswell, and honestly from my own experience of much higher quality than the ones on DT. The issue is that with TS2 being stable now and the release notes actively telling people that "typings from the typings may not be compatible with TS2, so use npm", people will only use these huge >1000 lines per file typings from DT, that share an issue and PR tracker with 2000 other declarations. And as people use these, they will submit PRs and contribute to the typings on DT, that will eventually get replaced by a redirect. I feel you should really prioritize this feature much much higher, it feels like you are just putting this at the end of the backlog as a "nice to have" while people are duplicating their efforts everyday. You promised to have full support for this scenario and now TS2 is out and it doesn't seem like anyone has even started working on this... Some clear statement or ETA would be really nice 😐

[RyanCavanaugh]

I apologize that this hasn't happened sooner. We're the same engineers as on TypeScript and as you can see we just shipped 2.0, so things have been pretty busy. We're also spending a lot of time just reviewing DT PRs and this has been taking longer than expected during the transition period.

We'd totally accept a PR on this if you'd be willing to help implement it. Otherwise I think we're looking at 4-6 weeks out at the earliest.

[felixfbecker]

It's been 6 weeks now, how is the progress?

[mhegazy]

We still have not looked at this. PRs are still welcomed.

[blakeembrey]

@mhegazy I think the only thing stopping this would be the structure. Has that been formalized yet? If it has, I think the only change required for very basic support would be adding git submodule update --init --recursive to https://github.com/Microsoft/types-publisher/blob/ca97be3faedd30bb33e9c02245c55fb50a6feaff/src/get-definitely-typed.ts#L13.

For the structure, we just need to know what we need to make it work. Is types-publisher using package.json yet? How do we specify files to publish (we can follow the current tsconfig.json pattern)? How is multiple versions supported - we can make repos follow whichever pattern and the publisher will just work the same as long as submodules automatically update to the latest commit. At a later date, I can build some additional tooling around submodules to make contributing easier.

[mhegazy]

if we end up using submodules, then if the new repo has the same shape, nothing needs to be done here. it should just work. the investment needed would be in getting a tool that runs tests and updates them. not sure what is the best experience here.

[felixfbecker]

then if the new repo has the same shape

what shape exactly?

the investment needed would be in getting a tool that runs tests and updates them

types-publisher doesn't need to run tests, the repos themselves all have Travis. I would imagine a typings publish command that uses the GitHub API to open a PR at DT. Which is the obvious disadvantage of submodules, someone needs to approve this PR. This hasn't been a problem in typings/registry so far though.

[blakeembrey]

@mhegazy I can write a bot that does the updates part, or we can create a CLI later. Either way, having something seems better than nothing at this point because we're getting a number of issues asking for @types to be published which we can't control.

As for the testing tool, what would that need to look like? Most of our repos have individual tests already, plus the registry runs a final structural test. I imagine when the PR is opened that DefinitelyTyped would run its own test suite and as long as that also supports submodules we should be in business.

[mhegazy]

types-publisher doesn't need to run tests, the repos themselves all have Travis.

no. we need one place to run tests to maintain consistency. say you change node definition in a separate repro, we need to run all tests against all declarations that depend on node in Definitely typed before pushing a package.

The test tool is not in the types-publisher, but on DT.

[mhegazy]

As for the testing tool, what would that need to look like? Most of our repos have individual tests already,

I meant a tool that runs on regular basis, updates submodules, and runs tests, and if so pushes them.

[mhegazy]

what shape exactly?

the shape of a typical folder in DT.

[blakeembrey]

If you add for support submodules wouldn't that work already - the tests should run on PR right?

I meant a tool that runs on regular basis, updates submodules, and runs tests, and if so pushes them.

See:

I can write a bot that does the updates part, or we can create a CLI later. Either way, having something seems better than nothing at this point because we're getting a number of issues asking for @types to be published which we can't control.

the shape of a typical folder in DT.

That's fine, but formalising this is important if things are going to become less centralised for contributors. Especially things like https://github.com/Microsoft/types-publisher/issues/3 which is important for us to maintain a matrix of versions for things like node.d.ts, we could either find ourselves maintaining multiple branches with multiple submodules or a single repo with folders contained, etc.

[mhegazy]

Either way, having something seems better than nothing at this point because we're getting a number of issues asking for @types to be published which we can't control.

Sorry if this seems obvious, but what is the urgency for this? and why not put it on DT?

[felixfbecker]

no. we need one place to run tests to maintain consistency. say you change node definition in a separate repro, we need to run all tests against all declarations that depend on node in Definitely typed before pushing a package.

That doesn't make much sense to me. You essentially want to run the tests of every single of the 2000 typings on every PR? That is like running the tests of every of the millions of modules that depend on lodash for a PR to lodash. When I change the node typings, it will result in a new version. As long as all other typings depend on the previous version, nothing breaks. Then humans can go in and update the definitions to adapt the latest API. Again, perfect analogy to a change in the lodash. You wouldn't require a PR to first change every dependent of lodash before merging the API change in. Or did I misunderstand you?

[mhegazy]

That's fine, but formalising this is important if things are going to become less centralised for contributors. Especially things like #3 which is important for us to maintain a matrix of versions for things like node.d.ts, we could either find ourselves maintaining multiple branches with multiple submodules or a single repo with folders contained, etc.

For #3, i would assume whatever works for a regular DT folder, will work for a mapped submodule. do you agree?

[mhegazy]

That doesn't make much sense to me. You essentially want to run the tests of every single of the 2000 typings on every PR

No. only on the subset of packages that depend on the one that changed. typically this is much smaller than 2000.

[mhegazy]

That is like running the tests of every of the millions of modules that depend on lodash for a PR to lodash.

I do not want to run it on all lodash dependencies. i want to run it on the ones we are publishing. that is much smaller.

[mhegazy]

As long as all other typings depend on the previous version, nothing breaks

they do not. they depend on "*"

[blakeembrey]

Sorry if this seems obvious, but what is the urgency for this? and why not put it on DT?

We have 220 definitions that we maintain in @types, not to mention other peoples, and DefinitelyTyped gives us zero control over the ability to review issues or PRs to the definitions. It would immediately create fragmentation. We also can not iterate very quickly, which can be important like when heavy refactoring to larger definitions like node.d.ts or rethinkdb.d.ts.

Most importantly, once something is put directly in DefinitelyTyped the author loses ownership. This might not seem like a big deal, but as an author it is. The side effect of losing the author as an owner means that no one really owns the definition, which means that when it comes time to do updates to the definition there's no one really pushing for it unless it's vital. I've seen this a number of times without outdated definitions that no one has bothered updating because no one owns.

[blakeembrey]

For #3, i would assume whatever works for a regular DT folder, will work for a mapped submodule. do you agree?

I think so, submodules should act exactly the same as part of the structure. The one thing we will likely need support for is package.json to specify our (type) dependencies. Has types-publisher been updated for that? (I can't recall what I saw around package.json).

Edit: Or do we need to follow double approaches - using references for DefinitelyTyped support and package.json for standalone tests?

[felixfbecker]

No. only on the subset of packages that depend on the one that changed. typically this is much smaller than 2000.

That means when we do a breaking change in the node typings, we cannot merge it until every single node-dependent in DT is changed? That won't scale. Why not let the typings express a strict dependency, instead of *?

[blakeembrey]

@felixfbecker I don't think that'll be a big problem for us, if the Express typings are wrong according to node.js we should definitely be updating them anyway.

@mhegazy Also, the reason we can't/won't copy into DefinitelyTyped is it just makes the process/workflow horrible. It means there's two possible places changes could be coming from. Every update we would need to 1. look in DefinitelyTyped, 2. copy back changes, 3. commit our change, 4. copy changes back to DefinitelyTyped. Going all in on DefinitelyTyped is not possible because of the previously mentioned reasons - it's unreasonable to expect an author to keep up with a mono-repo of thousands of definitions and unrelated issues just to maintain something. The project separation using GitHub affords us sanity, which allows us to do things like maintain separate issues on definitions, plan new features (e.g. https://github.com/types/npm-rethinkdb/issues/2) and use things like GreenKeeper to automatically know when something needs updating (e.g. https://github.com/types/npm-scmp/pull/1).

[mhegazy]

The one thing we will likely need support for is package.json to specify our (type) dependencies. Has types-publisher been updated for that? (I can't recall what I saw around package.json).

The publisher will read package.json if one exists, and will load dependencies and peerDependencies from it. not sure if this is what you meant though.

Edit: Or do we need to follow double approaches - using references for DefinitelyTyped support and package.json for standalone tests?

I am not sure i understand the question

[mhegazy]

That means when we do a breaking change in the node typings, we cannot merge it until every single node-dependent in DT is changed? That won't scale. Why not let the typings express a strict dependency, instead of *?

we did this for a while. the issue here is that means that every time there is a change in a package, you need to republish all packages that depend on it so that they have the latest version. otherwise they do not get the updated version.

Also this created a lot of conflicts, since the user installs node@6.0, and one of their dependencies depends on node@4.0 and another one depends on node@4.2.

The observation here is most declaration files do add. and most dependencies are on latest. we do have the ability to "freeze" the dependency on a specific version if needed. but has not been needed so far.

[mhegazy]

Would expanding the pool of users with push rights to DT be an acceptable solution here?

[mhegazy]

also pinging @andy-ms

[blakeembrey]

The publisher will read package.json if one exists, and will load dependencies and peerDependencies from it. not sure if this is what you meant though.

I think that's great.

Would expanding the pool of users with push rights to DT be an acceptable solution here?

For which? I don't think it solves all the reasons I don't push to DefinitelyTyped myself. It's just to hard to maintain. FWIW, I can already push to DefinitelyTyped, it's the management lifecycle which is impossible with DefinitelyTyped (can't manage issues well and can't keep up with any related PRs/issues as subscribing results in hundreds of unrelated issues a day).

[mhegazy]

Thanks for the explanation. my concern is fragmentation. Submodules should be fine here since we are really looking at one place when publishing, but makes it hard for users to search for issues, and get help.

[felixfbecker]

we did this for a while. the issue here is that means that every time there is a change in a package, you need to republish all packages that depend on it so that they have the latest version. otherwise they do not get the updated version.

This pattern has been working fine for typings though. They will just use their own version of the typings until they are updated, and thanks to using NPM this could be done fully automatically using Greenkeeper. It would run the tests and after merge can directly open the PR at DT with whatever tool we come up with (like typings publish).

Also this created a lot of conflicts, since the user installs node@6.0, and one of their dependencies depends on node@4.0 and another one depends on node@4.2.

Node is a special case though. Iirc TS2 now will resolve typings dependencies so that every declaration can have there own declaration dependency version of module x, as long as they are in external module format. Node however is a global declaration, it is not a module, it is an environment. As such, in typings, it has always been a "global dependency" that users need to install manually so there is only one in every project. With NPM it should be a peerDependency (side note: People already are putting @types/node as a dependency everywhere and it already broke my builds in several projects in patch releases, see my comment here https://github.com/Microsoft/types-publisher/issues/173#issuecomment-259067922). We cannot just assume that every declaration version of Node will work because the versions could have totally different API.

Thanks for the explanation. my concern is fragmentation. Submodules should be fine here since we are really looking at one place when publishing, but makes it hard for users to search for issues, and get help.

That's why I like the submodule idea over JSON files, because in GitHub you can click on a submodule and it will take you to the repository. Imo the current structure makes it hard to get help, because you share one issue tracker for 2000 declarations.

Would expanding the pool of users with push rights to DT be an acceptable solution here?

For merging our own updates to declarations in the types org, definitely.

[unional]

they do not. they depend on "*"

Doesn't this break versioning? i.e. packageA@1.2.3 depends on packageB@2.0.0 and packageC@3.4.5 depends on packageB@1.0.0.

I understand that you will run tests on DT to make sure they don't break. However, major version change does break things. That's why proper versioning is needed.

I have not used @types in TS2.0 enough so by understanding may be wrong.

Also, as @felixfbecker mentioned, we can use GreenKeeper for this purpose.

As for testing, simple compile test is not sufficient, IMO: https://github.com/typings/generator-typings#about-writing-tests-for-typings

If this is landed and https://github.com/Microsoft/TypeScript/issues/7661 is available, you can leverage tests in the source package, making testing a much easier effort.

[mhegazy]

Just a quick update on this issue.

Approach:

• Submodules seem to be what everyone likes so far, so likely go with submodules.
• No change to DT test/publish infrastructure seem to be needed;
• The expectation is whatever rules (folder structure, file headers, compiler version, style, etc..) apply to a DT folder, would apply to submodule linked package
• We may need to make some tooling updates to simplify the process, e.g. nightly update PRs, or automatic merges for submodule update PRs.
• Open question: what does licencing mean here?
  • Would every package have its own licence?
  • Can licences be conflicting?
  • Or should we require that all packages be under the same licence?

Timing

• There a couple of changes we would like to do first before enabling this change first; namely publishing different versions of a package : https://github.com/Microsoft/types-publisher/issues/3 and adding linter to the test infrastructure.
  • @andy-ms is working on the linter rules now, so we should have this wrapped up soon
  • publishing different versions is the next one.

[blakeembrey]

Open question: what does licencing mean here?

The TypeScript team could restrict accepted submodules to open licenses which I'm sure wouldn't be a problem. Something like MIT, ISC and Apache 2.0. Aside from that, taking the LICENCE file and license from package.json should suffice with these licenses. I would probably suggest a quick review from a legal team to make sure, maybe there needs to be a contribution agreement (but I'm sure that sort of thing could also apply to regular PRs) for republishing work onto various mediums (NPM).

[blakeembrey]

@mhegazy Three notes on the approach that I think should be considered:

1. Compiler version: coupling submodules to the compiler version will make it very difficult for DefinitelyTyped to update without changes to all submodules (when/if that happens). Probably best not to run tests on the submodules directly to avoid that
2. We're using package.json for the header metadata information - could that be resolved before looking for index.d.ts and trying to read the header? Most of our repos do not actually use index.d.ts because we follow the source repo style for deep require compatibility. Also, the version is already in package.json. Not to mention, we use package.json anyway for people who need to npm link or npm install during local updates that haven't synced back to DefinitelyTyped
3. Happy to follow the same style guidelines, but it needs to be published so modules can use it (we currently use https://github.com/typings/tslint-config-typings)

[DanielRosenwasser]

Just as an update, publishing different versions is the current task that @andy-ms is working on - as @mhegazy mentioned, this issue will be addressed following that.

[felixfbecker]

I spent a couple of hours trying to implement this, but couldn't find a way to do a recursive clone with node-git. The docs are just horrible. The closest thing I found was https://github.com/nodegit/nodegit/issues/925

Is there any reason we cannot simply exec a child process? It would be so much simpler and node-git really seems overkill for the simple use case of cloning a repository.

[felixfbecker]

Can a maintainer please comment on whether it would be acceptable to just use a git child process?

[ghost]

Sorry for the delay. I originally had tried using the git command line but had problems running it on Azure. I'll have to investigate.

[zackarychapple]

Bump. Ran into an issue today where I had to remove node from typings because a @types module I had pulled in @types/node. I would like to clean up typings all together but cannot because of this issue. @blakeembrey

[felixfbecker]

@andy-ms did you find out anything? What Azure setup is this running in? Is it a Docker container?

[felixfbecker]

Is there any way we could help to speed this up?

[RyanCavanaugh]

Apology

I really owe you guys an update of where we are on this. I'm sorry that I haven't been more communicative here. Discussions have been happening internally and we should have posted notes from these meetings here to keep everyone appraised of the situation.

Status

As it stands, we don't think we can accept redirects at this time. Let me outline the pros and cons to be explicit that we're thinking about all parties' interests here. I'll use "definition owners" to refer to people who might want to establish an external repo to develop a .d.ts file in isolation -- for example, node.d.ts is extremely high-impact and high-traffic, and in an ideal world really would have its own repo. "Us" / "we" refers to the TypeScript team / @types in general.

The primary problems faced by definition owners today, faced with contributing to DT instead, are:

1. Turnaround for PRs. The PR backlog is still too long despite us contributing almost an entire full-time engineer's worth of time on this.
2. Repo size. The repo is too big. It takes a long time to clone or do other git operations.
3. Scoping. There isn't a clear correspondence between issues on the issue tracker and the underlying definitions.
4. Control. Due to our accelerated PR review process, it's possible (though somewhat unlikely) that a badly-written commit might get merged into a file where "proper" definition owners would have rejected the PR
5. Communication bandwidth. Subscribing to DT is impossibly noisy in terms of notifications, but many interested parties do want to know what's going on with key definition files.

Now let me go over the things we're thinking about from the types-publisher / DT management side

1. Testing. It's critical that anyone installing from @types doesn't get definition files which have errors in them.
2. Global updates. We sometimes need to change a large number of definition files as an atomic operation. This can happen because of breaking changes (rare), infrastructure changes (common though hopefully less so in the future), renaming (somewhat common), compiler improvements (e.g. the "excess property" checks we added found a bunch of "bugs" in the test suites), or other unforeseen reasons (unknown unknowns...). We also want to sometimes enable new TSLint rules for the entire DT repo and fix them "everywhere" all at once - see Andy's PR history for examples here.
3. Invasive updates. There are also times when a non-TS/non-DT person needs to update many (but not "all") files in the repo. This scenario needs to work with

The main proposal on the table is having git submodules which point to specific commits of external repos, or something isomorphic to that. At face value, this would certainly solve the definition owners' problems, but there are some really bad trade-offs here. Let me review the main definition owners' problems and discuss what we think a good alternative is.

Solution: External repos, but no redirect

The model we'd like people to move toward is to have their own repo (either on GitHub or wherever else). A copy of the .d.ts file itself remains in DefinitelyTyped. This can be set up in whatever file layout the owners desire. When an update occurs, either a bot or a person sends a PR to DT to update the .d.ts file. A comment in the header of the .d.ts file can point to the external repo URL for human-reading purposes. Additionally we could have DT-bot warn when a PR changing one of these files doesn't match the source version in the originating repo, if that'd be useful.

This enables some great new scenarios as well -- maybe the external repo doesn't actually have a .d.ts file checked in, but is instead the output of some more complex build process, for example.

Why not just have external repos give commit rights to the DT org? This doesn't solve the "invasive updates" scenario for non-MS/non-DT people who need to make wide changes.

Addressing problems

Let's review the problems, for both sides, listed above, and how this proposal affects them.

Turnaround

When a hypothetical external repo updates, they'll have to submit a PR to DT anyway to move the submodule pointer forward. This PR will end up with the same latency problems as the PRs today, so that problem doesn't actually get solved. Now you might say: we should set up a bot to automatically merge these PRs, and that's a great idea. But this only fixes PR turnaround for external repos.

Instead: what we really want to do is have a well-defined set of "trust" rules that would allow for automatic merging of ✔CI PRs. This could take the form of a per-folder OWNERS.md which lists people whose PRs can be merged as soon as CI passing. We'd have to be defining the set of trusted people implicitly by picking the "true" owning repo for a .d.ts file anyway.

Repo Size

A 580 MB repo isn't great to work with. But for the sorts of definitions people are talking about moving out, this problem doesn't go away. Anyone making changes to a .d.ts file needs to make sure they don't break any dependents of the file, and the only (reasonable) way to know which files depend on you is to have the entire repo so that the test runner can find them.

Instead: external repos can optimistically accept PRs and run the tests only when pushing a PR to DT. This is basically the same situation you'd be in with submodule redirects anyway.

Scoping

This gets solved equally well whether the external repo is a submodule or not.

Control

I think we generally get the best of both worlds here. We absolutely have to be able to make atomic updates to the entire DT repo, but having an external repo with the authors' "official" copy makes clear where those updates are coming from.

This is an explicit trade-off: External repo owners will have to be capable of taking "reverse" merges from DT from time to time.

Communication

This gets solved equally well whether the external repo is a submodule or not.

Testing

As stated above, everyone needs to be testing the "entire world" when making definition file updates.

Global Updates

With submodule redirects, global updates would be nearly impossible. Giving commit rights to everyone who wants to make global updates is a decent mitigation, but we'd have to somehow enforce that this was set up correctly at all times. As stated above, we're introducing a different cost ("reverse" merges from DT to the external repo), but we consider this to be preferable given the relatively small number of external repos we expect to exist.

Invasive Updates

This is the tricky one that isn't solved by giving commit rights to DT people. A reasonably common occurrence is that someone needs to change the internal structure of an export = target so that it can be correctly augmented in other modules. If the thing being refactored is in a submodule and has externally-hosted dependencies, this becomes an absolute nightmare - the author would have to send a PR to the main module, then a separate failing PR to the dependent module, then once both of those merge, she sends a submodule update PR to DT. But between when the second and third PRs merge, neither the first nor second module can send a submodule update to DT. And somehow you'd need to communicate the fact that the projects are in this state while it's all occurring so that the changes don't get treated like a normal build break and reverted.

Summary

I hope this outlines that we've thought carefully about this. The externally-hosted repo solution should fix a great majority of the problems presented by the monorepo, while still allowing good testing, refactorability, and predictability for the DT maintainers.

[blakeembrey]

I don't entirely agree. You're solution sounds basically like "copy and paste" when we could use git submodules for the identical process. The solution you propose, as a result, fixes neither the issues definition authors face nor does it solve issues that you face. I can see the compromises you're trying to make, but I don't feel they are the entirely right ones.

For instance, testing is something you can already achieve. This doesn't resolve any communication confusion and will continue to result in communication issues when users don't know where to log issues. The ability for these "global updates" to occur now means that external authors need to somehow be able to pull the reverse changes (which you did note). However, as you also noted, this may be difficult for authors to do as the DefinitelyTyped structure is very particular and might not match the repo (e.g. build tools).

There's a few little things, that if you're going down this road, would be good to see fixed. The NPM package would need to have the correct repo URLs, bugs URL, etc (pointing to the main repo and not DefinitelyTyped). An automated publishing tool to DefinitelyTyped and "bundler" (to make it possible for multi-file and non-index.d.ts input sources to work in DefinitelyTyped).

All in all, some assumptions you make such as "we consider this to be preferable given the relatively small number of external repos we expect to exist" seem incorrect. Even Typings, which effectively "lived" for less than a year and was never "official" compared to DefinitelyTyped, has almost 400 repos which is around 15% of DefinitelyTyped. Also, in the "invasive updates" section you seemed to explain how regular dependencies work "as a nightmare". I understand the want to not have to submit cascading PRs, but in reality that's rarely an issue (how often is a wrong definition depended on down the line) and is also easily solved by following correct semver.

I'd like to see some of the solutions that will automate this for maintainers, because at this point (10 months after the issue was brought up) and with no communication, I don't have enough faith that even with this proposal that there'll be enough movement without members from @typings building it - and we don't have enough information to know what it is you want built. At this point, I'd also like to give away ownership of @types because I'm not sure I'd be able to maintain it in this future.

[unional]

Another issue with the proposed solution is it does not seem to take versioning into consideration at all.

Your proposal is combining all typings in DT and consider it as one single version. That's not how npm and package works.

When you say that "We absolutely have to be able to make atomic updates to the entire DT repo", that should be a huge red flag.

UPDATE: I also notice that @types does not seem to solve version conflicts either. This issue comes up in SO several times and the situation will just get worse if it is not addressed.