Have version 4 been abandoned?

[craigchandler]

No updates the Azure.CosmosDB (v4) for 2 years

[j82w]

The v4 work is currently being planned for mid to end of 2022 but can still possibly change. @kirankumarkolli correct me if I'm wrong.

The v4 SDK was created as a preview to get feedback on the Azure SDK contracts which were being developed at that time. As part of the v4 preview it showed that the amount of work required to be fully complaint is not trivial. The v3 SDK was also just released a few months prior to the Azure SDK guidelines were created.

[kirankumarkolli]

@craigchandler our current recommendation is to use V3 SDK for all usages.

We might start working on V4 forward later this year, also please note that we expect V4 GA will take longer into next year. https://github.com/Azure/azure-cosmos-dotnet-v3/discussions/2601 is currently tracking the high-level work that needs to be done to land V4. We will keep updating that with latest updates/plans.

[petro2050]

@j82w @kirankumarkolli

Is full-fidelity available yet? I haven't seen any announcements on this. Is anything preventing it from being part of v3?

[j82w]

Full fidelity work is still in progress. When it becomes available it will be supported by the v3 SDK.

[petro2050]

@j82w is there a way we could track the status of this work? any epics/stories/etc.? I didn't see anything under the Projects tab.

[j82w]

@petro2050 it can be tracked here: https://github.com/Azure/azure-cosmos-dotnet-v3/issues/2994, https://github.com/Azure/azure-cosmos-dotnet-v3/issues/2995, https://github.com/Azure/azure-cosmos-dotnet-v3/issues/2996

[petro2050]

@j82w thanks for putting those out there. Is there any ETA for completion? I'm surprised FF is still in triage even though it was announced a long time ago.

[j82w]

@philipthomas-MSFT is there an ETA?

[mumby0168]

Hi all 👋

I help maintain the Cosmos Repository Pattern. I was wondering about some of the details of the V4 release.

1. Are they're going to be pre-release packages of this at any time?
2. I have seen there is a big push to switch to System.Text.Json for serialization. Is this going to be implementing a custom CosmosSerializer or will this be a full implementation losing the CosmosSerializer?
3. Where is best to keep up to date with the new V4 SDK? I was hoping to get a head start on our library at once the version has matured a little.

Thanks in advance 😃

cc @IEvangelist

[IEvangelist]

Yes, I would love to learn more about the v4 release and echo @mumby0168 here. Thank you all.

[IEvangelist]

Hi all 👋

I help maintain the Cosmos Repository Pattern. I was wondering about some of the details of the V4 release.

1. Are they're going to be pre-release packages of this at any time?
2. I have seen there is a big push to switch to System.Text.Json for serialization. Is this going to be implementing a custom CosmosSerializer or will this be a full implementation losing the CosmosSerializer?
3. Where is best to keep up to date with the new V4 SDK? I was hoping to get a head start on our library at once the version has matured a little.

Thanks in advance 😃

cc @IEvangelist

Hi @mumby0168 - I asked the team on a separate thread. They pointed to this discussion which tracks all the work to be done for v4. As of right now, it's not looking like it's going to happen any time soon.

https://github.com/Azure/azure-cosmos-dotnet-v3/discussions/2601

[xDDDx]

Hi, I have also a question about V4 features in V3 SDK.

we are still on .NET Core 3.1 and stopped upgrading to .NET 6 because of not fully supported System.Text.Json in the current V3 SDK. We want to remove Newtonsoft Dependency from our project, if we switch to .NET 6, but it's not possible yet.

The biggest issue we found yet is, that the queries are not correct, if we are using JsonStringEnumConverter.

Is it on the road to implement all the features to fully support System.Text.Json in V3 SDK?

Thanks in advance

[Mortana89]

For those looking at V4 SDK; two years ago we made the switch and jumped on V4, because of STJ and getting rid of Newtonsoft. V4 is lacking many 'new' features and has been dead for a very long time, so didn't take long before we jumped to v3 (we went from legacy to V4 and then V4 to v3). A custom cosmos serializer is all that is needed to support STJ in v3! You do need some reflection magic if I recall correctly, but again, don't let STJ push you to V4, there's really no use in V4, we learned the hard way 😉

[TechWatching]

@Mortana89 Good to know indeed. My concern is that every Azure SDK is now adopting the new Azure SDK standards with common patterns and libraries except Cosmos DB SDK. I like very much the new Azure SDKs because everything is easily testable, I can configure them easily in the same place and they all look alike. It is what I would want for the Cosmos DB SDK.

[bartelink]

For me the bottom line is that there will eventually be a V4, but there's no benefit in tying yourself to its schedule - there's plenty information shared about the intentions (and the fact that there is not a date set as such) in the open issues in this repo.

I've linked to some minimal F# code snippets in this issue, but the bottom line is that the V3 SDK has a sample for how to plug in a custom serializer hook that uses STJ instead.

When the time comes, one would hope that the V4 API layout lets that shim be deleted, but until then you want to be on the maintained SDK codebase that's receiving bug and perf fixes. It may well be that V3 and V4 are both ultimately delivered out of this repo as different API surfaces on the same core thing.

[xDDDx]

The problem is not the serializer. STJ serializer works well, but the LinqToSQL conversion (ExpressionToSQL.cs) is hardcoded to search for Newtonsoft cenverter, so it does not work, if you have something like this

var query = from m in MyCollection
                   where m.EnumProperty == MyEnum.Foo
                   select m;

and you use JsonStringEnumConverter.

I think the best sollution for us at the moment is to upgade to .net 6 and let the SDK working with Newtonsoft and define all our models with both type of converters. So Newtonoft and STJ together.

Not the best solution for me, but the only one reasonable at the moment

[bartelink]

Ah, OK - your assertions and workardound make sense then. I believe there are issues logged regarding that but its worth verifying your problem and/or repro scenario is logged and/or bumping that; I was taking the question at face value as discussing the general merits of and timelines of a shift to V4.

Obv v4 branch is definitely behind on such bugfixes so there is zero chance of it working better, and its extremely likely that any real future v4 will use the same logic so best to get that logged and fixed if it is not already. While V4 may provide a very good story and/or default of STJ out of the box, I think its safe to assume that it will continue to support Newtonsoft.Json too, i.e. be based on extending the existing code and tests as opposed to doing a rewrite.

[jcocchi]

We have no committed plans to release a GA v4 SDK so I'm closing this issue. Please continue to use the v3 SDK for now, and as always if you have any feature requests or bugs feel free to open a new issue.

You can track our progress towards v4 here #2601

[JayAtSherweb]

We have no committed plans to release a GA v4 SDK so I'm closing this issue. Please continue to use the v3 SDK for now, and as always if you have any feature requests or bugs feel free to open a new issue.

You can track our progress towards v4 here #2601

Jan 18th 2022 was the last update there, so it seems to be very much abandoned then?

[mberdev]

A summary : - CosmosDb SDK V4 is dead - V3 requires you to write custom serializer if you want to use System.Text.Json instead of Newtonsoft. It's not super hard but it still takes hours of your precious development time to avoid crashing silently on corner cases such as "what happens when the document is null, what happens when it's empty, what happens when one of the fields has the wrong type or nullability, etc." - V3 does not work with Linq as soon as you use System.Text.Json (as explained above). If you think it does, let me stop you right there. It doesn't. The iterators are broken and return empty sets when you least expect it. You'll need to write all your SQL queries as strings.

I am baffled that Microsoft keeps advertising Cosmos as if it was a real product when in reality the codebase to use it in 2023 is rotting away -- or, some might say, doesn't exist.

[ealsur]

@mberdev Thanks for your candid feedback.

As @jcocchi clarified in April, there are no plans to GA V4 at this point. V4 was an experimental version released many years ago and was not pursued. As with experiments, sometimes they yield a result that is continued, some other times they don't. The Nuget packages for V4 SDK are not listed anymore.

Regarding your well defined points:

V3 requires you to write custom serializer if you want to use System.Text.Json

This is correct, you can use a custom serializer to replace Newtonsoft.Json with System.Text.Json or any other serializer technology. We have a usable example implementation: https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos.Samples/Usage/SystemTextJson/CosmosSystemTextJsonSerializer.cs - if you have any feedback or believe it can be improved, please create a new Issue and we can work together to improve it. Null or empty documents is not something that exists in Cosmos DB as far as I am aware (but you can have operations with no body in the response), and field type and nullability is something that Json decorators in System.Text.Json can handle, but I might be wrong.

V3 does not work with Linq as soon as you use System.Text.Json

This is true and there are multiple Issues already tracking this. This is not just for System.Text.Json, but any custom serializer. The truth is that this point is not related to V4 though. We could release V4 tomorrow that this issue would still be present. Please use the other existing Issues if you want to add more constructive details to those discussions that can help the team working on it.

the codebase to use it in 2023 is rotting away -- or, some might say, doesn't exist.

I am honestly confused by this statement, this code base has daily activity and monthly releases of the V3 SDK with improvements, fixes, and new features. We are open to any feedback that will help with communications of new releases (today it's done through the changelog). Please create a new Issue with your feedback and we can work together to address it.

[Barsonax]

Reading this and the fact that the cosmosdb emulator is broken and seemingly unmaintained makes we wonder if cosmosdb is going to get dropped in the future? This doesn't give me much trust...

[FabianMeiswinkel]

@Barsonax - Cosmos DB is not getting dropped. The main reason for not moving forward with a V4 SDK right now is that there is still a large portion of our customers, who still use the V2 SDK. The V2 SDK is going to retire in August (not being supported anymore -not even for security updates) - right now focusing on moving them to the V3 SDK before introducing yet another major version is necessary. Regarding emulator - it is still supported - last update has been pushed in March of this year. If you provide me your constructive feedback what is missing or where you face challenges I can redirect it to the right folks, who work on the emulator. you can use fabianm AT microsoft.com via email. Thanks, Fabian

[Socolin]

Regarding emulator - it is still supported - last update has been pushed in March of this year. If you provide me your constructive feedback what is missing or where you face challenges I can redirect it to the right folks, who work on the emulator. you can use fabianm AT microsoft.com via email. Thanks, Fabian

Thanks for those explanations

I just have some feedback about the emulator.

I also consider it as broken too: The last time I tried the emulator on linux, the emulator was unusable (After some time it of using 100% of CPU for nothing and being non responsive). I reported the problem with all the info I had to the email on the emulator page and never got any feedback on this...

Would it be possible to open a Github repo (like with just a Readme) to report those issues in public, it would be easier to track problems and know if a problem is already known or if there is a need to take time to report.

[bartelink]

edited - there's a repo, and the issue list is maintained relatively well, with responsiveness e.g. https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/54#issuecomment-1828171843

I had mentioned two things I run into here, but sketching them here is pointless - if I want to get them logged, I need to go there, see if they're already logged etc. But, looking at the first page of the issues list there, it's hard to feel that it's the primary input channel...

[Barsonax]

@Barsonax - Cosmos DB is not getting dropped. The main reason for not moving forward with a V4 SDK right now is that there is still a large portion of our customers, who still use the V2 SDK. The V2 SDK is going to retire in August (not being supported anymore -not even for security updates) - right now focusing on moving them to the V3 SDK before introducing yet another major version is necessary. Regarding emulator - it is still supported - last update has been pushed in March of this year. If you provide me your constructive feedback what is missing or where you face challenges I can redirect it to the right folks, who work on the emulator. you can use fabianm AT microsoft.com via email. Thanks, Fabian

Hmm ok but still does not give much confidence since the emulator is broken. A working emulator is essential for tests. We cant and won't run everything in Azure.

There have been many ppl complaining about the emulator through issues on the emulator GitHub already. It sort of works but is unstable on windows (503's) and completely broken on Linux (doesn't seem to start properly). For instance this issue https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/56.

Even on windows running parallel tests against it will break the emulator. I could only get it stable when running tests single threaded.

[bartelink]

@barsonax I throw plenty load at it in docker on an Intel Mac (Apple Silicon is not yet supported) with considerable parallelism - while it does not have good perf with parallelism (no speedup beyond a few workers), I would not say it's unstable, and I don't believe any specific load can get it into a mangled state.

That said, my experience winds up very similar - I perpetually have warm knees and/or fan sounds as the emulator never survives a restart (so I have wound up with great tools and scripts for establishing known states, which is no bad thing...).

In terms of emulation not being workable, the JSON PATCH update APIs like 500ing a lot (and they don't seem to ever succeed, no matter how many retries for specific documents (given a specific state and operation))

[sajeetharan]

@Barsonax I throw plenty load at it in docker on an Intel Mac (Apple Silicon is not yet supported) with considerable parallelism - while it does not have good perf with parallelism (no speedup beyond a few workers), I would not say it's unstable, and I don't believe any specific load can get it into a mangled state.

That said, my experience winds up very similar - I perpetually have warm knees and/or fan sounds as the emulator never survives a restart (so I have wound up with great tools and scripts for establishing known states, which is no bad thing...).

In terms of emulation not being workable, the JSON PATCH update APIs like 500ing a lot (and they don't seem to ever succeed, no matter how many retries for specific documents (given a specific state and operation))

Hi @bartelink thanks for providing the feedback, we have a dedicated emulator repo to report the issues and feedbacks. You can either post here https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/54 or start a new thread.

[bartelink]

@sajeetharan Yes, see https://github.com/Azure/azure-cosmos-dotnet-v3/issues/2976#issuecomment-1860929403

I don't have a proper repro worthy of the name for the PATCH stuff, thought its definitely consistent.

I really doubt there there's anything I can do about defining the parameters re it hanging (it just freezes on container 8-14 of 16) and having to be repaved - it's unimaginable to me that it's not simply a known issue that is just not going to make the top of the list any time soon.

While part of me would like to hope the repaves can stop, I also would like to get to a Silicon Mac. But it's as likely I'll wind up using dynamodb-local via https://github.com/jet/equinox...

[mberdev]

the codebase to use CosmosDb in 2023 is rotting away (SDK3) -- or, some might say, doesn't exist (SDK4).

I am honestly confused by this statement, this code base has daily activity and monthly releases of the V3 SDK with improvements, fixes, and new features. We are open to any feedback that will help with communications of new releases (today it's done through the changelog). Please create a new Issue with your feedback and we can work together to address it.

I am confused by your confusion. SDK 4 remains experimental, while SDK 3 forces you to choose between Linq and System.Text.Json. - If you want to use System.Text.Json then you have to give up on Linq (because, as it has been explained, in the SDK 3 Newtonsoft is hard-coded in some places, which doesn't work well with the custom serializer required to use System.Text.Json, which causes empty iterators when Linq gets involved) - If you want to use Linq then you have to give up on System.Text.Json (same reason as above, in reverse)

In one of Microsoft's star products (Azure CosmosDb), can we honestly say that a codebase that forces you to either hand-write your queries as plain strings (because, no Linq) or use an "obsolete" serializer (because, stuck with Newtonsoft) can be considered "healthy" by 2023 standards? Especially after a year of silence about SDK4, which is never announced clearly as potentially coming out of the experimental stage? This sounds very, very, VERY suspicious from the outside.

I am not creating a separate issue, because we are right on-topic here : "Have version 4 been abandoned?" (or, read the subtext: "Has Microsoft given up on bringing CosmosDb SDK to Microsoft's own 2023 standards? And if so what does it mean for CosmosDb?" )

[bartelink]

Especially after a year of silence about SDK4, which is never announced clearly as potentially coming out of the experimental stage?

Unfortunately there are multiple issues covering V4 in this tracker, but to be fair, communication in general re V4 has been prompt, clear and consistent. There was a period in 20/21 where things were confusing and not well communicated (and people like me were overoptimistic about how likely it was to become a maintained entity alongside, and soon replacing, V3)

Ultimately, if you're on a team like this SDK team, taking a dep on Azure.Core has pros and cons - who would not want to be in line with the general trend/masterplan; however, not breaking people and/or causing them churn is also important. It's far easier to do that if you're a relatively boring and trivial client.

There is ongoing STJ work - I agree it has taken ages and is way out of whack with Newtonsoft EOL IRL and STJ adoption. I for one would not underestimate the ugliness/awkwardness of trying to land a change as thorny as providing in the box STJ+NSJ support in this repo - they can't just go in and rip and replace. (Someone will hopefully dig out the PRs)

Having said that, I don't have STJ serialized things I'm trying to LINQ over at the present time, so I can only imagine your frustration. And I'm definitely looking forward to ripping out https://github.com/jet/equinox/blob/master/src/Equinox.CosmosStore/CosmosStoreSerialization.fs#L34-L48 and glorying in that being under an the power of an SEP field.

maintainers: It would be nice if there was a project / roadmap somewhere that reflects in flight issues and priorities that someone like @mberdev could be referred to...

[JayAtSherweb]

We're hitting all the same issues mentioned by @mberdev as well.

Has not been fun to say the least.

As I mentioned before, long long long over do....

[kirankumarkolli]

Custom serializer LINQ support: https://github.com/Azure/azure-cosmos-dotnet-v3/commit/4d9b7bd9848ddde59a8aea50e2dff4c25045f83d addressed this gap with a new contract. This is getting shipped part of next preview release 3.39 preview (will be out by end-of-month).