Board Discussion: Multi-version support, Avro support

[kurtzeborn]

Storage is the first track 2 library to cover this ground, but all libraries will need to do this soon. We want to discuss the best path forward to support multiple versions of the service API in the client and appropriate ways to fail in each language when the version requested does not support the SDK entrypoint being used.

Second topic (same meeting please): We want to discuss how the new storage feature support for QuickQuery and ChangeFeed can be implemented within the track 2 guidelines while supporting a service API implementation which returns Avro content. QuickQuery is coming in STG73. ChangeFeed is not versioned and support is being added now.

[KrzysztofCwalina]

@seanmcc-msft, I have heard you prototyped/explored alternative ways of doing the version checks, including EnsureVersion helpers that could be called within service method bodies. Do you still have branches with the prototypes? I think such prototypes would be good to look at when we evaluate the pros and cons of the options.

cc: @tg-msft

[tg-msft]

@seanmcc-msft - Could you also point out the operation level checks for snapshots that were problematic in Track 1? It would be helpful to get some context around that as well.

[tg-msft]

@JonathanGiles, @johanste, @bterlson, @JeffreyRichter - Could you please take a quick look at Avro support for Java/Python/JS/Go before this discussion? It would be super helpful to understand options across all languages.

[johanste]

For Python, there are several packages available. The "official" packages are(avro and avro-python3). The former claims support for Python 2.7 and 3.5+ in its package metadata, but the description says "bindings for python 2" in some places. The latter only supports python 3.4+. Based on a cursory glance, they are both pure python packages. Purely based on release history, the projects seem to be maintained.

Neither state external dependencies, and it doesn't seem like they have binary dependencies/dependencies on other libraries being installed on the machine (e.g. libavro-dev).

There are additional packages that claim to be faster (e.g. fastavro) - but that does require a binary extension.

[adrianhall]

Has AVRO support gone through API review / Office of the CTO review? It seems to me that this is a reasonable expectation so that services that have similar requirements all converge on the same solution, which would make it easier to contemplate from an SDK perspective.

[bterlson]

For JS, there seems to be two reasonable options:

avsc

npm: https://www.npmjs.com/package/avsc github: https://github.com/mtth/avsc downloads: 25k/week last release: 1 month ago dependencies: 0

avro-js

npm: https://www.npmjs.com/package/avro-js github: https://github.com/apache/avro (monorepo of many language bindings) downloads: 0.5k/week last release: 5 months ago. dependencies: 1 (underscore 🙁)

Remarks

avsc is vastly more popular and seems actively maintained. Avro-js has the benefit of coming from a well-known organization that we have a history of working well with and having bindings for many different languages. I would lean toward avsc however since it's more broadly used.

[JeffreyRichter]

For Go:

· Linked-In has an Avro packagehttps://github.com/linkedin/goavro last updated October 2019, 577 stars, 131 forks. Apache license (but Linked-In is MS now so not sure if/how this impacts things). This package depends on Google Snappy for (de)compression support.

· https://github.com/go-avro/avro was last updated on 12/2017. Apache 2 license. 38 stars, 18 forks. I’d avoid this package due to lack of interest/maintenance.

This is really about it; I’m surprised to see the lack of Go support for Avro. If we need to do this, then I think the Linked-In (and Google Snappy) package is our best bet.

• Jeffrey Richter (watch Architecting Distributed Cloud Appshttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fazure.microsoft.com%2Ftraining%2Fdistributed-cloud-apps%2F&data=02%7C01%7Cjeffreyr%40microsoft.com%7Cc2139bebacc8433bd13308d76a014af0%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637094426089210301&sdata=GjJbhammA5C%2F1Pve%2FXjbMq8wjjpGr2W1KSrYaCmzq7E%3D&reserved=0)

From: Brian Terlson notifications@github.com Sent: Friday, January 24, 2020 2:10 PM To: Azure/azure-sdk azure-sdk@noreply.github.com Cc: Jeffrey Richter jeffrichter@live.com; Mention mention@noreply.github.com Subject: Re: [Azure/azure-sdk] Board Discussion: Multi-version support, Avro support (#969)

For JS, there seems to be two reasonable options:

avsc

npm: https://www.npmjs.com/package/avsc github: https://github.com/mtth/avsc downloads: 25k/week last release: 1 month ago dependencies: 0

avro-js

npm: https://www.npmjs.com/package/avro-js github: https://github.com/apache/avro (monorepo of many language bindings) downloads: 0.5k/week last release: 5 months ago. dependencies: 1 (underscore 🙁)

Remarks

avsc is vastly more popular and seems actively maintained. Avro-js has the benefit of coming from a well-known organization that we have a history of working well with and having bindings for many different languages. I would lean toward avsc however since it's more broadly used.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Azure/azure-sdk/issues/969?email_source=notifications&email_token=AARLJP7AO6GQYKFXNALLA3TQ7NRL3A5CNFSM4KK72PI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ4H3NA#issuecomment-578321844, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AARLJP5EUQQYUQWCGE5ZVL3Q7NRL3ANCNFSM4KK72PIQ.

[JonathanGiles]

The only option I could find for Java is Apache Avro. It seems widely used (all spring tutorials even made use of it), and relatively frequently updated. It has dependencies on Jackson (which is fine, although it does lag behind the version we use), and Apache commons-compress (which I would rather we avoid but that is unreasonable). We would ideally see a release of Avro with an updated Jackson dependency - we might want to file an issue and contribute a patch that does this.

[tg-msft]

I implemented a proof of concept Changefeed client at https://gist.github.com/tg-msft/d45b5e0351f0d668d2fb578df625b0fb using only System.Text.Json and Azure.Storage.Blobs. It can parse enough Avro to support Changefeed scenarios in about 200 lines of sloppy prototype code.

[adrianhall]

Recording: https://msit.microsoftstream.com/video/38251aaa-a391-4ad5-b3e8-f3d6fe62deaa

• Service versioning
  • Checking service versions via pipeline.
  • What storage chose to do:
    • Pipeline policy grabs the service version header, if same, then ok. Throw an ArgumentException if not supported.
    • It's really modular, and may be supported in Azure Core in future.
  • On our side, support multiple service versions.
    • Proposal: We should be doing a client-side check for compatibility with the requested service.
    • should this be a case-by-case basis?
    • In storage, service ignores headers. (But there are problems with this right now - to be investigated)
  • Controversy: How should we implement the checks?
    • Concerns with proposal (pipeline policy):
      • Sooner or later, it's going to be a very complex piece of code, with little understanding.
      • Compatibility with future versions
    • Storage is meant to be a temporary fix.
    • Storage code is "not that complex" - not most elegant, but modular.
    • Using a policy has one place it will fail - any SDK call that makes multiple REST API calls.
    • Do we want to put any limits on model construction? There would be additional problems with model round-tripping
  • How did we end up with the pipeline being the focus?
    • As you start to have more and more overloads, you have to make sure you are carrying it in all the right places.
    • The right place to put it is in the generators? It's not clear - belief is correct.
  • KC believes the best place (most transparent, etc.) is close to the user code. In the policy is too far away.
    • We can make the generators do the checks, but that should be the safety net.
  • Requirement is that all the checks are documented, and the public documentation is associated with the user code.
  • Proposal: if it doesn't cause meaningful harm, then don't block it. Checks would be best effort.
    • If it's meaningful, then let's mitigate them in the API layer.
  • Proposal: Do checks in PUT/POST/PATCH/DELETE (i.e. mutations)
    • It seems a reasonable proposal to PM (Storage)
    • Johans comments about docs is a concern as well.
  • Does the contract has an issue with deprecation? There are mechanisms / designs for this?
  • Are we going to be in the scenario where branches are used for service versions?
    • At some point, service versions will be deprecated. In practical terms, not while it is being used.
    • The question is "how many service versions does the client library support?"

Follow up: #1 Peter Marino - will return to Storage and scope down to the POST/POST/PATCH/DELETE.

2 Anna Tant - will tell Peter Marino which APIs fail to fail.

DECISION: Storage will go back and scope it to "valuable and correct" instead of relying on the policy / Azure Core.

• AVRO Support
  • What is AVRO? It's a wireline format that is valid as long as you have complete objects - it's a streaming format.
  • Will be used in storage for changefeed, and quick query? Used in event hubs for writing to blobs. Used also for SPARK, Hadoop, etc.
  • Also have AVRO support in stream analytics and azure data analytics.
  • What is the problem? We don't have parsers for AVRO .NET - MS AVRO and Apache AVRO - they both look like an intern project, with no docs and doesn't comply with dependency guidelines. JS - Apache AVRO (intern project), plus another more-well-used. It's another dependency on a community project. Go - two libraries (one by LinkedIn, updated in 2017, the other was abandoned) Python - Apache (1 py2&3, the other py3) they seem to be slow. Others use binary extensions. Java - Apache AVRO - been around for 10 years, very heavily used (does actually have an API listing) - there is a cost in terms of dependency mgmt.
  • Why not one of the other messaging protocols? They want to use this for replication scenarios (monitoring deltas).
    • First changefeed was cosmosdb, you can wire up to azure functions / event grid, and pump / process those changes.
    • Change notifications on storage does this.
    • So this is not targeting live scenarios.
  • Changefeed schema changes over time, so needs work.
  • Is there a case for building an AVRO parser? Is this ITOps or developer experience? (PM-Storage will ask)
  • Do we want programmatic APIs (since it's really about quick query) Quick Query is trying to bring some sort of compute to the data. SQL like (subset) query with the GET call, run the data that matches the query. Unknown as to whether AVRO is needed there - it might be that it was just there. They also want to support transforms. It's basically GET BLOB with added information to indicate the query (binary response)
  • The query language is not part of AVRO at all.
  • Writing a simple AVRO parser in 4-5 languages is doable, but needs coordination with storage team to ensure they don't add features.
    • Cost of doing this is not trivial.
    • What are our options? The libraries are not inspiring.
    • Managing dependencies would bring more cost.
  • Is it possible to ask for content negotiation, so we can ask for JSON or AVRO.
  • Compete (AWS)
    • We modelled the events in Coral (AWS version of swagger) - get data over wire, wrote our own parsers.
    • Did not hear of AVRO, but sounds very similar.
  • Given other services have taken a dependency on AVRO - e.g. big data - it's about getting it into a format for big data processing.
  • Proposal: parse the wireline format, but don't expose that it is AVRO. Don't take a dependency.
    • Treat schema as a breaking change. For quick query, specifically, but also changefeed.
    • Go back to service team and formalize schema.
  • AVRO has schema evolution - but that's something we want to lock down.
  • Data is streamed back.
  • If we had a number in mind of customers, how many applications owuld actually write code against this? For quick query, we believe strategically that intelligent object storage is valuable, and customers will want it.
    • It's a compete with AWS
    • What does the API for Quick Query look like?

Follow-up: #1 Peter Marino - figure out if we can get out of changefeed AVRO support.

2 Peter Marino - write small minimal parsers based on quick query parser.