API v3

[fculpo]

Hi,

I saw you are working on API v3 compatibility, do you plan to use a specific branch to contribute on, or are you keeping the work private ?

[jjmaldonis]

So for now we're going to keep the work private, not because we don't want people to see it, but more so because we are starting completely from scratch and haven't figured out the detailed implementations yet. There are some issues with the current Cass, e.g. you can't easily make calls to different regions, and the database stuff also leaves something to be desired. We also came up with a totally new way to store/access the data from Riot (which is super cool).

So because the new Cass is so much different, we are starting from scratch and can't just improve the current version of Cass. And because we are still figuring out implementation details, things are changing significantly and quickly (for the last couple weeks, Rob and I have been spending hours and hours talking about things 3-5 nights of the week).

When we get a working framework finished and have some documentation written, we'll likely make it public, at which point people will want to help and we'll have some info to point them at to explain all the complicated stuff that's going on. But until we have something that's worth showing people, we're hesitant to make it public. I hope that makes sense?

[fculpo]

Well it is understandable, but opening it may also drag attention (in a positive manner) and you may get valuable input.

Your call anyway. Do you plan to have v3 API working by the end of the deprecation notice ?

[jjmaldonis]

You're totally right that we are leaving out valuable input like yours and I totally get where you're coming from. It sucks, and it's not something I want to do, but the alternative is summarizing hours and hours worth of discussions and explaining why a bunch of different example implementation code is good/bad with significant detail. That alone would take 1-2 hours every night, and who knows, it could all change the next day.

We absolutely take our users' input to heart, and in fact that's what's driving many of our changes. If I had an efficient way of explaining what we are doing, I'd do it in a heartbeat. Also when we make the new version of Cass public, that certainly does not mean it is finished or that it can't undergo some significant change to make it better. Feedback will still be crucial at that stage.

We will absolutely have the v3 API working by the end of the depreciation period.

A couple teasers for what's coming and some ideas for you to think about:

for match in Summoner(name="Kalturi").matchlist.search("Annie"):
    if "Rod of Ages" in match["Annie"].items:
        # process the match

There's a lot to process in the above code, and the syntax for that logic might look slightly different, but we want it to be incredibly easy and intuitive to search for data inside objects (like matches).

cmasteries = Summoner(name="Kalturi").champion_masteries
above_20k = cmasteries[lambda cm: cm.points > 20000]  # returns all values of `cmasteries` where the callable inside the getitem returns True

In addition to the code examples above, you'll be able to download images from data dragon, have an intuitive and easy way to create databases (e.g. in-memory cache, mongo, etc) that will be queried before asking Riot for the data, and the type system is revamped to make it easier to use than ever (e.g. the above examples).

If you have opinions on that stuff let us know, and if you think of some cool functionality that you'd be interested in let us know too. The sky's the limit.

[jjmaldonis]

Actually, can you explain to me what this statement seems to do?

for match in Summoner(name="Kalturi").matchlist.search("Annie"):

[fculpo]

Thanks for the quick answer. Do you plan redis for in-memory (with persistence option, which would be awesome) ?

for match in Summoner(name="Kalturi").matchlist.search("Annie"):

I guess it iterates on all matches where summoner Kalturi played Annie :)

[robrua]

Hey @fculpo

There will be both pure-python and redis-based in-memory caching options. Right now the target list for Caches/DBs to support on or shortly after release of the new version is: Pure-python in-memory, Redis, Memcached, Mongo, SQLAlchemy.

Hopefully with the new data storage API that's in the works it should be really easy for both the cass devs and the community to create new data stores for cass. You can get a sneak peek at the library we're working on to underpin the new databasing stuff @ https://github.com/meraki-analytics/datapipelines

[tylerturk]

It would be useful if in the new version there was a way to flag a debug mode for whichever auto-databasing mechanism there was to avoid editing the installed cassiopeia files directly. For my current use case, I just set echo=True for the SQLAlchemy connection so I can trace what's going on.

[jchristgit]

That looks really promising.
I'd really love if it was available on GitHub so I could maybe help with something, but then again, it's up to you.

[fculpo]

Hi, do you have any insight on the api v3 compatibility ?

[jjmaldonis]

It will not be backwards compatible if that's what you mean.

[fculpo]

Sry i wasn't clear ;) I would like to know the ETA for api v3 release.

Le 12 juin 2017 15:33, "Jason Maldonis" notifications@github.com a écrit :

It will not be backwards compatible if that's what you mean.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/meraki-analytics/cassiopeia/issues/103#issuecomment-307790474, or mute the thread https://github.com/notifications/unsubscribe-auth/AAT1_WOPOEJoldWB0yPKtLhyGhjBfGN-ks5sDT4xgaJpZM4NLNZq .

[jjmaldonis]

The ETA is ASAP! We are working on it and have 90% of the hard work done. Now we need to go through and implement all the endpoints. The type system, rate limiter, and some of the other stuff are finished. There is a bit more to do though.

I'm hoping to have a version of it on a github branch at some point in the near future, although I'm hesitant to give a specific date. We'll put it on github once we are confident nothing will be changing.

Since we also need to update Orianna, there are some things that won't be entirely finished at release. For example, we likely won't have time to do all the doc strings, write good examples, etc. It's also likely that we will only support a couple databases at the beginning, and users can implement their own as they need them.

This has been a ton of work.

[pfmoore]

Do I need to plan for the possibility that there will be a period when the old APIs have been retired, but the new version of cassiopeia isn't available? I'm doing some data collection for an analysis project, and I'm wondering whether I need to do a quick slurp of "everything I can think of" ASAP, to avoid getting blocked due to lack of access to data over the transition.

I understand that it's not easy to provide firm dates, and I don't want to sound like I'm putting pressure on you. I'm perfectly OK with working with whatever best guesses you have at this point.

And thanks for all the work you put in on the project - I've only just started playing with the riot API, and discovering this library turned what was looking like a long and frustrating task into a positive joy :-)

[jjmaldonis]

There's almost zero chance we don't get Cass done before July 24th. I'll be surprised if it isn't usable by July 1st. It may not have all the databasing functionality completed, but if you're just saving the json to disk then it certainly won't be an issue.

Glad you enjoy using Cass too!

[pfmoore]

Fantastic, thanks for that :-)

[robrua]

Hey everybody,

Since we're getting closer and closer to the deprecation period for the old APIs, @jjmaldonis and I want to get something you all can look at and start coding against soon; we're just wrapping up most of internals of the new cass and getting started building out the API layer. Our plan is to do a "soft-release" where we upload the new codebase to a new public branch by Sunday, June 25.

We're confident we can get all the essentials finished by then, though we'll probably be lacking on documentation, tests, and maybe some peripheral features like support for certain popular databases. Hopefully we can find some maintainers at that point to help us build those things out and to help maintain the project going forward.

Because of the expected lack of documentation, we're planning on setting up a discord-based VOIP soft-release event on Sunday, June 25, at 3:00PM EDT (New York Time), during which we can go over the new architecture & features, as well as get feedback from anyone interested. Let us know if you guys think that's a waste of time or if you'd want to be involved. If it seems like people are interested, we'll provide more details as we get closer to that date.

Best, Rob

[fculpo]

Hi,

Sorry for the late answer, I think it's great ! I hope to be able to join the discord (i'm on CEST time, so about 9pm here).

Keep up the work !

[fculpo]

Also I really hope Redis to be part of the initial release. I saw high perf improvements since i use my redisCache implementation (which could greatly be improved)

[fculpo]

Unrelated, but do you plan (or already implemented) to have some mecanism to automatically return cached items if Riot is unavailable (aka 503 or like)

I guess this would be like:

• cache every Riot response in database (json format if possible)
• ask Riot for new data when requested
• if 503, return cached data if available (with possibly an additional field stating that this data is cached)

[jjmaldonis]

Redis will be in the initial release, time permitting. An in-memory Cache, Redis, SQLAlchemy, and Mongo are the top priorities in that order I think.

Unrelated, but do you plan (or already implemented) to have some mecanism to automatically return cached items if Riot is unavailable (aka 503 or like)

This may not be entirely automatic if you have a database behind the Riot API, but as long as you take some care to update it with the data you want in it this will absolutely be possible.

Rob wrote a "data pipeline" which you can find in one of our other repos. The data pipeline is the series of data storage packages that either provide or store data. For example, a data pipeline might be: 1) an in-memory cache, 2) an SQL database, 3) the Riot API. In this example, if an object is loaded, it will first look in the in-memory cache for the data it needs; if it doesn't find it in the cache, it will look in the database, then it will hit the Riot API if the database doesn't have what you asked for. Then, once the datapipeline found the correct data, it will back-track and update all the datastores with that new data (ie if the data was pull from the Riot API, the database would be updated followed by the in-memory cache).

For your use case, you could have another database as # 4 (behind the Riot API). Then if the Riot API returns a 503 and fails to return the data, the datapipeline would just look in your database because it's the next thing in the list. You may need to manually dump what you want into that database, however, because it won't automatically be updated every time data is pull from the Riot API.

I'm 90% sure that's how it works, and Rob can likely give more details on Sunday too.

[robrua]

Hey all,

Here's an invite link for a discord we've started to have a more chat-oriented place to discuss the project: https://discord.gg/8fjUbfa

It'll be where we do the soft-release gig at 3:00PM EDT (New York Time) on the 25th.

[pfmoore]

Any updates? Looks like the V3 API is the only one available now, but there's no new release on PyPI. Is the v3-development branch stable enough to use yet?

Edit: Actually, as it doesn't have a setup.py, it's not installable, so I guess the answer is no...

[jjmaldonis]

The v3-dev branch is probably not stable enough to use. Champions, Items, Summoners, and Champion Masteries work as of late last night. We ran into some unexpected issues as the last piece of functionality (the cache) was being added on Saturday morning that led to a need to rewrite 90% of what was in place. Since Saturday morning I've spent every spare minute on Cass and it's beginning to come together. Most of the functionality works now but it needs to be rewritten for all the different endpoints and data. Unfortunately this is just going to take time to change everything, but it's not for lack of effort. We've been working on this for months and months and expected to easily hit the deadline.

So sorry for the delay, but you can expect to be able to use Cass within the next few days. Documentation, examples, and testing will come after that. We may not make a release on PyPI until we have some docs and examples in place, partly so that people have something to reference when they see the changes in the new release. Luckily we have a few people testing and helping to find bugs already, but the release will certainly not be bug free. For the next few weeks I suggest that you pull the code directly from github rather than use PyPI because bug fixes will likely occur often.

[pfmoore]

Thanks for the update - no rush, I can easily hold off on my project for a while. I appreciate all the effort you're putting into it - thanks :-)

[jjmaldonis]

Ok everyone, Cass is pretty much done now. See the v3-development branch, which will shortly be merged into master.