search

NuGet / NuGetGallery

NuGet Gallery is a package repository that powers https://www.nuget.org. Use this repo for reporting NuGet.org issues.
https://www.nuget.org/
Apache License 2.0
1.54k stars 644 forks source link

Prevent db2catalog from skipping entries #3991

Open loic-sharma opened 7 years ago

loic-sharma commented 7 years ago

The db2catalog (formerly feed2catalog) job occasionally skips entries due to concurrency issues between reads and inserts in the feed. This causes packages to be missing from the registration and flat-container.

Currently, db2catalog keeps track of the greatest created, edited, and deleted times it has processed. Whenever the job pulls more entries from the feed, it filters out all packages whose operation time is before the greatest processed time. This is incorrect as new feed entries may appear on the feed in a different order than the time of their operation. Instead, the db2catalog should pull all packages that are near the greatest processed created, edited, and deleted times, and then should only filter out packages that have already been processed.

Related to #3613 and #3614

skofman1 commented 7 years ago

On hold. Pending potential architectural changes

joelverhagen commented 5 years ago

This is happening a lot while repo-signing is happening. Instances found recently:

NetPack.Rollup 0.3.0-alpha0058 ServiceGovernance.Repository.Models 0.1.3 OrgChartJS 3.2.9 MassTransit.AmazonSQS 5.1.5.1665-develop CoreDdd.Rebus.UnitOfWork 3.2.0 TWCore.Reflection 2.1.179-alpha7 CatUtils 1.2.0 GivePenny.GherkinSpec.TestAdapter 0.1.3

I have manually reflowed all of these.

joelverhagen commented 5 years ago

More:

Bridge.Fundamentals 2018.12.6.2 DCSDK.DataSource.HttpClient 1.10.2 Momentum.Pm.PortalApi 5.14.511-beta Olive.BlobAws 1.0.24

joelverhagen commented 5 years ago

The monitoring issue is here: https://github.com/NuGet/Engineering/issues/318

joelverhagen commented 5 years ago

More:

Beyova.AspNet 1.4.2 PureActive.Queue.Hangfire 1.0.175-prerelease

joelverhagen commented 5 years ago

More:

AWSSDK.DAX 3.3.1.28

joelverhagen commented 5 years ago

More:

Love2dCS 11.0.18

joelverhagen commented 5 years ago

More:

DotVVM.Templates 2.1.0-preview01-final Magick.NET.Web-Q16-HDRI-x86 7.9.2 Microsoft.Azure.IoT.Edge.Function 3.1.0

joelverhagen commented 5 years ago

More:

BeeHive.Azure/3.0.0-alpha3

joelverhagen commented 5 years ago

More:

digital-flare.oxy.pooling.pool/2.15.6 Dynamicweb.FilePublisher/1.0.3

scottbommarito commented 5 years ago

Another one (caught by Db2Monitoring): PragmaticWorks.LegiTest.Ssis2016.Runtime/2019.3.2.718-beta002

joelverhagen commented 4 years ago

Another one (https://github.com/NuGet/Engineering/issues/3095): Sentry.Protocol 2.1.2-beta