Stable API (strict) migration issue

[espjak]

When using stable API (https://www.mongodb.com/docs/manual/reference/stable-api/) in strict mode, we run into trouble with the migration:

 ---> Hangfire.Mongo.Migration.MongoMigrationException: Migration failed in Hangfire.Mongo.Migration.Steps.Version09.CreateSignalCollection
 ---> MongoDB.Driver.MongoCommandException: Command create failed: BSON field 'create.size' is not allowed with apiStrict:true..

Hangfire.Mongo v1.10.4

Is there any plan on supporting stable API?

[gottscj]

Hi @espjak,

I didn't know about the stable API. Let me try it out.

Thanks

[gottscj]

@espjak,

I cant reproduce, could you please share your config, I used:

var mongoUrlBuilder = new MongoUrlBuilder("mongodb://localhost:27017/?readPreference=primary")
                {
                    DatabaseName = "hangfire"
                };
                var uri = mongoUrlBuilder.ToMongoUrl();
                var settings = MongoClientSettings.FromUrl(uri);
                settings.ServerApi = new ServerApi(ServerApiVersion.V1);

                var mongoClient = new MongoClient(settings);

                var storageOptions = new MongoStorageOptions
                {
                    MigrationOptions = new MongoMigrationOptions
                    {
                        MigrationStrategy = new MigrateMongoMigrationStrategy(),
                        BackupStrategy = new CollectionMongoBackupStrategy()
                    },
    SlidingInvisibilityTimeout = TimeSpan.FromSeconds(5),
   CheckQueuedJobsStrategy = CheckQueuedJobsStrategy.Poll
};

//config.UseLogProvider(new FileLogProvider());
config.SetDataCompatibilityLevel(CompatibilityLevel.Version_180);
config.UseColouredConsoleLogProvider(LogLevel.Trace);
config.UseMongoStorage(mongoClient, mongoUrlBuilder.DatabaseName, storageOptions);

thanks

[gottscj]

@espjak,

You could also try to set "SupportsCappedCollection" to false. But above config successfully completes the reported migration without setting the "SupportsCappedCollection" flag to false.

[gottscj]

@espjak, I am able to reproduce using the "strict" option. You need to use following settings:

var storageOptions = new MongoStorageOptions
{
    MigrationOptions = new MongoMigrationOptions
    {
        BackupStrategy = new NoneMongoBackupStrategy()
    },
    CheckQueuedJobsStrategy = CheckQueuedJobsStrategy.Poll,
    SupportsCappedCollection = false
};

You might be able to use the default "CheckQueuedJobsStrategy" setting "Watch" if you are using a cluster, but I didnt check. Also I need to make an adjustment to the MonitoringApi as the stable API does not support the "distinct" method. I can do a $group instead. I will patch as soon as possible

[espjak]

The provided settings seems to work (except the dashboard/MontioringApi as stated). As we have been using capped collection before, do we have to do anything now that we set SupportsCappedCollection to false?

Thanks for quick feedback!

[gottscj]

@espjak,

I would expect that wouldnt work for the stable API as this will require setting a max size of the collection, which was original issue?

Setting SupportsCappedCollection to false means hangfire.mongo will poll for queued jobs

Thanks

[espjak]

It seems we can use "Watch" as well, along with SupportsCappedCollection = false, which is a better fit for us. In the comments it says, "Will still poll using 'QueuePollInterval', what does this mean? It also states this with "TailNotificationsCollection", but we have not noticed any extra polling going on with it, at least not to the extend that the "Poll" strategy does.

Do you have any ETA on a fix for the MonitoringApi distinct issue?

[gottscj]

@espjak,

Yes you can use watch. the capped collection is mainly for single node mongo instances with the need for low latency enqueuing. "Watch" is the default setting.

MongoDB will still poll for enqueued jobs even though you use settings "Watch". But only if no jobs are enqueued and the semaphores are not signaled.

I plan to release this weekend

[espjak]

With the new release we can use stable API (strict) with "Watch" and SupportsCappedCollection = false. We also had issues with migrations and expected null values, but i see that there is already commited a fix for this in master :)

Thanks for the help!