`types-registry` seems to be missing packages in the latest version

[moltar]

I noticed @types/semver in particular, but there are others too.

Repro:

npm init -y

npm i types-registry@0.1.665
cat node_modules/types-registry/index.json | jq -r '.entries | keys | sort_by(.) | .[]' > 0.1.665.txt

npm i types-registry@0.1.666
cat node_modules/types-registry/index.json | jq -r '.entries | keys | sort_by(.) | .[]' > 0.1.666.txt

diff 0.1.665.txt 0.1.666.txt

⬇️ Diff

Diff

``` 59a60 > addons-linter 159d159 < amqplib 315d314 < array-normalize 322,323d320 < array.prototype.flat < array.prototype.flatmap 328d324 < asap 411d406 < aws-serverless-express 439d433 < babel-plugin-tester 473d466 < badwords 485d477 < barnard59-base 547d538 < big-rat 610d600 < body 662d651 < bresenham 683d671 < browser-update 724d711 < business-rules-engine 749d735 < call-bind 949d934 < clownface 974d958 < collections 1044d1027 < config 1152d1134 < country-flag-icons 1165d1146 < crc 1198d1178 < crypto-js 1233d1212 < currency-formatter 1254,1255d1232 < cypress-image-snapshot < cypress__code-coverage 1268d1244 < d 1354d1329 < date-now 1369d1343 < db-stations 1407d1380 < degit 1490d1462 < django-bananas 1535d1506 < domexception 1625d1595 < eases 1699,1700d1668 < ember-data < ember-data__adapter 1702d1669 < ember-data__serializer 1712,1714d1678 < ember__application < ember__array < ember__component 1718d1681 < ember__engine 1722d1684 < ember__object 1726,1727d1687 < ember__routing < ember__runloop 1731d1690 < ember__test 1780d1738 < error-cause 1805a1764 > eslint-ast-utils 1810a1770 > eslint__eslintrc 1841d1800 < event-emitter 2014d1972 < feathersjs__errors 2026d1983 < fetch-headers 2044d2000 < fibers 2139d2094 < flux 2175d2129 < format-unicorn 2209d2162 < fs-extra 2227d2179 < function.prototype.name 2684d2635 < gauge 2759d2709 < git-clone 2799,2800d2748 < gl-vec3 < gl-vec4 2829d2776 < glur 2894d2840 < graphql-upload 3035d2980 < gulp-uglify 3041d2985 < gun 3098d3041 < has-package-exports 3101d3043 < has-symbols 3143a3086 > hex-to-binary 3182d3124 < hot-formula-parser 3260d3201 < hydra-box 3267d3207 < hyphen 3280d3219 < iarna__toml 3379d3317 < integrate-adaptive-simpson 3387d3324 < intersects 3844,3845d3780 < jsonexport < jsonfile 4134d4068 < lapo__asn1js 4136d4069 < lasso 4237a4171 > libhoney 4716d4649 < markdown-it-emoji 4732d4664 < marko 4772d4703 < mdx 4807d4737 < mertasan__tailwindcss-variables 4877d4806 < mime 4900d4828 < mithril 4925d4852 < mochawesome 4940d4866 < module-alias 5015d4940 < mssql 5022d4946 < muicss 5103a5028 > nearest-color 5113d5037 < nes 5133d5056 < next-nprogress 5301d5223 < nodejs-captcha 5429d5350 < observ 5531a5453 > package-json-validator 5601a5524 > passport-42 5641d5563 < passport-saml 5698d5619 < penner 5867d5787 < posterus 5926d5845 < prismjs 6015d5933 < psd 6084d6001 < ramda 6138d6054 < rdf-loader-code 6148d6063 < rdfjs__environment 6151d6065 < rdfjs__fetch-lite 6153d6066 < rdfjs__namespace 6156,6157d6068 < rdfjs__prefix-map < rdfjs__score 6163,6164d6073 < rdfjs__term-map < rdfjs__term-set 6166d6074 < rdfjs__traverser 6267d6174 < react-dev-utils 6274d6180 < react-dom 6361d6266 < react-instantsearch 6363d6267 < react-instantsearch-dom 6366d6269 < react-is 6392,6393d6294 < react-loadable < react-loadable-visibility 6451d6351 < react-native-crypto-js 6551d6450 < react-native-vector-icons 6560d6458 < react-native__assets 6579d6476 < react-phone-number-input 6584d6480 < react-plotly.js 6600d6495 < react-reconciler 6605d6499 < react-refresh 6752,6754d6645 < rebass < rebass__forms < rebass__grid 6788d6678 < redux-auth-wrapper 6813d6702 < redux-persist-transform-encrypt 6844d6732 < reflexbox 6852d6739 < registry-auth-token 6857d6743 < relay-runtime 6883d6768 < request-promise-native 6990d6874 < route-cache 7111d6994 < scheduler 7159d7041 < secure-random-uniform 7163d7044 < secure-weighted-sample 7176d7056 < selenium-webdriver 7204d7083 < semver 7231d7109 < serverless 7334d7211 < simple-websocket 7357d7233 < siren-parser 7425d7300 < socketcluster 7455d7329 < source-map-support 7460d7333 < sparql-http-client 7465d7337 < spdx-expression-parse 7566d7437 < strange 7575d7445 < stream-json 7627d7496 < strophe 7634d7502 < styled-components 7693d7560 < svg-sprite-loader 7747d7613 < synchronous-autocomplete 7775d7640 < tape-promise 7819d7683 < testing-library__cypress 7862d7725 < thunky 7864d7726 < tiff-to-png 7928d7789 < token-introspection 7988d7848 < tsc-watch 8087d7946 < unicode-trie 8157d8015 < use-sync-external-store 8171d8028 < util.promisify 8175d8031 < uuid 8177d8032 < uuid-browser 8263d8117 < viz.js 8269d8122 < voca 8279d8131 < vscode-notebook-renderer 8415d8266 < webpack-virtual-modules 8446d8296 < whatwg-url 8468d8317 < win-ca 8588d8436 < xmpp__client 8598d8445 < xmpp__iq 8602d8448 < xmpp__resolve 8609,8610d8454 < xmpp__starttls < xmpp__stream-features 8639d8482 < yargs 8681d8523 < zen-observable ```

[moltar]

And just to be clear, @types/semver does not seem to be unpublished, and is still available on npm, and the semver package itself does not distribute types.

https://www.npmjs.com/package/@types/semver

https://www.npmjs.com/package/semver?activeTab=code

[sandersn]

Trying to repro this locally and semver is still there. I'll see if there are any other differences compared to 0.1.665

Edit: Nope, the local run looks the way it should.

[moltar]

❯ cat node_modules/types-registry/index.json | jq -r '.entries | keys | sort_by(.) | .[]' | grep semver
git-semver-tags
is-semver
semver-compare
semver-compare-multi
semver-sort
semver-stable
semver-utils

[jakebailey]

I have also run it locally, both on a clone and on a downloaded tarball like in CI, and it isn't missing these packages. Very odd. Maybe if we just rerun the CI job, this will go away?

[moltar]

Here's running inside Docker to really eliminate any environmental issues:

docker run --rm node:buster sh -c \
    "apt-get -qq update && apt-get -qq -y install jq && cd /tmp && npm -s i types-registry@0.1.666 && cat node_modules/types-registry/index.json | jq -r '.entries' | grep semver"

  "git-semver-tags": {
  "is-semver": {
  "semver-compare": {
  "semver-compare-multi": {
  "semver-sort": {
  "semver-stable": {
  "semver-utils": {

[jakebailey]

Sorry, I think we weren't clear; we are actually running the tool that produces the package in dry-run mode and looking at its output; the published package is definitely missing things.

[moltar]

Could there be a race condition, perhaps, that only triggers in low-resource environments, such as CI?

[jakebailey]

I just reran it and no change, but in rereading everything it turns out there is some caching locally and removing it got me to a reproduction:

rm -rf packages/definitions-parser/data packages/publisher/output
GITHUB_ACTIONS=1 pnpm run --filter @definitelytyped/publisher parse
GITHUB_ACTIONS=1 pnpm run --filter @definitelytyped/publisher publish-registry --dry
# check packages/publisher/output

[jakebailey]

With that in mind, #771 fixes the problem, so this feels like some weird package loading bug.

[moltar]

Last time I looked the code in this repo, I saw that some of the work was done in the worker processes, so perhaps it is something related to that?

[jakebailey]

It's most likely the fact that this job doesn't actually pnpm install the DT repo now that it's a real monorepo. That screws with the loading process because we too eagerly check for errors, which I'm guessing aren't propagated up somewhere accidentally and so we're publishing incomplete data (whereas it entirely failed other jobs that we've already fixed up). #771 fixes this by not doing those error checks up front, with #769 removing the need to do analysis entirely.

[sandersn]

771 indeed fixes this. 0.1.668 has the correct packages as far as I can tell from diffing against 0.1.665