Potential issue with npm `LossyFeedEvent` & versioning

[tom--pollard]

For a given snapshot of the npm endpoint https://registry.npmjs.org/-/rss the assumption that entries are in chronological order to their respective pubDate doesn't seem to be valid. As can be seen below ngx-jira-issue-collector has a pubDate of Tue, 02 Jun 2020 08:37:26 GMT nestled between the 7th May 2021 dates, and other occurrences can be seen.

This presents an issue when applying the findOverlap logic for the lossy logging, as this relies on using the oldest pkg of any given polling set as a marker for triggering the event.

npm rss snapshot

``` https://www.npmjs.com/ RSS for Node Fri, 07 May 2021 12:17:52 GMT Fri, 07 May 2021 12:17:52 GMT 600 https://npmjs.com/package/@100mslive/sdk-components https://npmjs.com/package/@100mslive/sdk-components Fri, 07 May 2021 11:56:20 GMT https://npmjs.com/package/ngx-jira-issue-collector https://npmjs.com/package/ngx-jira-issue-collector Tue, 02 Jun 2020 08:37:26 GMT https://npmjs.com/package/@learnthis/nest-cqrs https://npmjs.com/package/@learnthis/nest-cqrs Fri, 07 May 2021 12:17:33 GMT https://npmjs.com/package/mio-common-js https://npmjs.com/package/mio-common-js Fri, 07 May 2021 12:17:28 GMT https://npmjs.com/package/html-converter-full https://npmjs.com/package/html-converter-full Fri, 07 May 2021 12:17:28 GMT https://npmjs.com/package/@cognite/sdk-wells https://npmjs.com/package/@cognite/sdk-wells Fri, 07 May 2021 12:17:23 GMT https://npmjs.com/package/droff https://npmjs.com/package/droff Fri, 07 May 2021 12:17:25 GMT https://npmjs.com/package/@imtf/rjsf-conditionals https://npmjs.com/package/@imtf/rjsf-conditionals Fri, 07 May 2021 12:17:24 GMT https://npmjs.com/package/@laufire/utils https://npmjs.com/package/@laufire/utils Fri, 07 May 2021 12:17:18 GMT https://npmjs.com/package/eslint-plugin-ghost https://npmjs.com/package/eslint-plugin-ghost Fri, 07 May 2021 12:17:16 GMT https://npmjs.com/package/terrapin-test-1 https://npmjs.com/package/terrapin-test-1 Fri, 07 May 2021 12:17:10 GMT https://npmjs.com/package/@crystaldesign/diva https://npmjs.com/package/@crystaldesign/diva Wed, 07 Apr 2021 07:50:17 GMT https://npmjs.com/package/@crystaldesign/diva-react https://npmjs.com/package/@crystaldesign/diva-react Wed, 07 Apr 2021 07:50:17 GMT https://npmjs.com/package/@crystaldesign/diva-framework https://npmjs.com/package/@crystaldesign/diva-framework Wed, 07 Apr 2021 07:50:11 GMT https://npmjs.com/package/starstuff-style https://npmjs.com/package/starstuff-style Fri, 07 May 2021 12:17:05 GMT https://npmjs.com/package/@native-html/transient-render-engine https://npmjs.com/package/@native-html/transient-render-engine Fri, 07 May 2021 12:17:05 GMT https://npmjs.com/package/@bugsnag/js https://npmjs.com/package/@bugsnag/js Tue, 06 Apr 2021 09:39:46 GMT https://npmjs.com/package/@bugsnag/electron https://npmjs.com/package/@bugsnag/electron Mon, 26 Apr 2021 13:54:31 GMT https://npmjs.com/package/abstract-type https://npmjs.com/package/abstract-type Sun, 06 Mar 2016 13:59:29 GMT https://npmjs.com/package/@bugsnag/node https://npmjs.com/package/@bugsnag/node Tue, 06 Apr 2021 09:39:38 GMT https://npmjs.com/package/@bugsnag/plugin-electron-renderer-event-data https://npmjs.com/package/@bugsnag/plugin-electron-renderer-event-data Mon, 26 Apr 2021 13:53:58 GMT https://npmjs.com/package/@bugsnag/plugin-stackframe-path-normaliser https://npmjs.com/package/@bugsnag/plugin-stackframe-path-normaliser Fri, 07 May 2021 12:16:48 GMT https://npmjs.com/package/@bugsnag/plugin-angular https://npmjs.com/package/@bugsnag/plugin-angular Tue, 06 Apr 2021 09:38:41 GMT https://npmjs.com/package/@bugsnag/plugin-electron-app https://npmjs.com/package/@bugsnag/plugin-electron-app Mon, 26 Apr 2021 13:53:51 GMT https://npmjs.com/package/ad-kernel https://npmjs.com/package/ad-kernel Fri, 07 May 2021 12:16:38 GMT https://npmjs.com/package/smartapp-automator https://npmjs.com/package/smartapp-automator Fri, 07 May 2021 12:16:38 GMT https://npmjs.com/package/tailwindcss-language-service https://npmjs.com/package/tailwindcss-language-service Fri, 07 May 2021 12:16:37 GMT https://npmjs.com/package/@native-html/css-processor https://npmjs.com/package/@native-html/css-processor Fri, 07 May 2021 12:16:35 GMT https://npmjs.com/package/stocache https://npmjs.com/package/stocache Fri, 07 May 2021 12:16:23 GMT https://npmjs.com/package/rnjs-core https://npmjs.com/package/rnjs-core Fri, 07 May 2021 12:16:11 GMT https://npmjs.com/package/whistle-sdk https://npmjs.com/package/whistle-sdk Fri, 07 May 2021 12:15:40 GMT https://npmjs.com/package/stylelint-variable-check-teamix https://npmjs.com/package/stylelint-variable-check-teamix Fri, 07 May 2021 12:15:59 GMT https://npmjs.com/package/rn-bottom-test https://npmjs.com/package/rn-bottom-test Fri, 07 May 2021 12:15:47 GMT https://npmjs.com/package/@xflr6/chatbot-engine https://npmjs.com/package/@xflr6/chatbot-engine Fri, 07 May 2021 12:15:44 GMT https://npmjs.com/package/bimplus-renderer https://npmjs.com/package/bimplus-renderer Mon, 26 Apr 2021 12:54:22 GMT https://npmjs.com/package/@flowplayer/player https://npmjs.com/package/@flowplayer/player Fri, 23 Apr 2021 13:16:36 GMT https://npmjs.com/package/ublo https://npmjs.com/package/ublo Fri, 07 May 2021 12:15:13 GMT https://npmjs.com/package/homebridge-ewelink https://npmjs.com/package/homebridge-ewelink Fri, 07 May 2021 05:36:53 GMT https://npmjs.com/package/tinkiet https://npmjs.com/package/tinkiet Fri, 07 May 2021 12:15:11 GMT https://npmjs.com/package/@xflr6/chatbot https://npmjs.com/package/@xflr6/chatbot Fri, 07 May 2021 12:15:07 GMT https://npmjs.com/package/@gapizza/eslint-config https://npmjs.com/package/@gapizza/eslint-config Fri, 07 May 2021 12:15:00 GMT https://npmjs.com/package/cra-template-timiodulate https://npmjs.com/package/cra-template-timiodulate Fri, 07 May 2021 12:14:52 GMT https://npmjs.com/package/@switchboard-xyz/switchboard-api https://npmjs.com/package/@switchboard-xyz/switchboard-api Fri, 07 May 2021 12:14:47 GMT https://npmjs.com/package/react-native-set-wallpaper https://npmjs.com/package/react-native-set-wallpaper Fri, 07 May 2021 12:14:47 GMT https://npmjs.com/package/schummar-translate https://npmjs.com/package/schummar-translate Fri, 07 May 2021 12:14:41 GMT https://npmjs.com/package/dejquery.co https://npmjs.com/package/dejquery.co Fri, 07 May 2021 12:14:35 GMT https://npmjs.com/package/@vitalyrudenko/dependency-registry https://npmjs.com/package/@vitalyrudenko/dependency-registry Fri, 07 May 2021 12:14:28 GMT https://npmjs.com/package/honeyfield-component-library https://npmjs.com/package/honeyfield-component-library Fri, 07 May 2021 12:14:27 GMT https://npmjs.com/package/@jolie/jpm https://npmjs.com/package/@jolie/jpm Fri, 07 May 2021 12:14:20 GMT ```

[Qinusty]

Looking at @bugsnag/electron in the snapshot above it seems that the pubDate in this rss feed matches the latest tag whereas it is included in the updates feed since there is a NEW version published with label next 17 minutes ago (from time of writing this).

The fetchVersionInformation() function grabs the latest tag from the package specific api endpoint currently which is indeed a problem if that isn't the version we've been notified of via the firehose feed.

This raises a separate issue of old npm packages being published via package-feeds when a next version is updated and we publish the latest which could be from several weeks ago.

[tom--pollard]

In reverse of that ngx-jira-issue-collector pubDate matches that of it's dist-tags previous release-8.x, and latest is in fact the newest tag https://www.npmjs.com/package/ngx-jira-issue-collector?activeTab=versions

"dist-tags":{"latest":"9.0.3","release-8.x":"8.0.1"}

So we'd (when latest is the actual tag that updated) end up publishing the 'correct' latest version, however the created_date would be that of the older dist-tag. This is appose to (when latest isn't that tag that updated) the @bugsnag/electron example where the version and create_date are correct for latest, all be it for an old release which may have already been tested and which is not the version that caused the entry.

There is a "time" field provided:

time":{"created":"2020-05-13T12:30:02.074Z","9.0.0":"2020-05-13T12:30:02.334Z","modified":"2021-05-07T12:17:47.640Z","9.0.1":"2020-05-13T13:12:59.225Z","8.0.1":"2020-05-13T15:21:02.565Z","9.0.2":"2020-06-02T08:37:26.504Z","9.0.3":"2021-05-07T12:17:45.447Z"

We could assume the last entry in the node is that of the event which triggered the rss entry, and use that as the version & created_date? A caveat there is if the same package is included multiple times in a single polls results, as both would resolve down to the same version.