[Feature] Crest API - Githubissues

zardada commented 9 years ago

Will the CREST API be implemented? Mainly so we can get rid of the IGB and use out of game browser to track our in game movements.

Zumochi commented 9 years ago

None of the required headers that you receive from using the IGB have been implemented in CREST yet. :disappointed:

dagoaty commented 8 years ago

The authenticated CREST endpoint https://crest-tq.eveonline.com/characters/<CHAR_ID>/location/ now returns a location if the character is online.

{ "solarSystem": { "id_str": "30000142", "href": "https://crest-tq.eveonline.com/solarsystems/30000142/", "id": 30000142, "name": "Jita" } }

raphendyr commented 8 years ago

Ping... To my knowledge we currently should have character location and waypoint/destination methods.

I think we should start by implementing SSO auth and then requesting CREST token.

Someday the IGB is going away anycase. Also there is currently the CREST app contest. Dunno if the ews can take part on that or not.

raphendyr commented 8 years ago

@dagoaty any idea how long that endpoint is cached? updated on every request?

raphendyr commented 8 years ago

When running EWS e.g. on tablet, there could be enpoint/page that would have signature submit form that would update signature list of current system. Idea is that one would use the mapper from tablet, but copy-paste signatures to igb or external browser.

dagoaty commented 8 years ago

@raphendyr

The endpoint is cached for 10 seconds. There is currently no way to know to start polling the endpoint, i.e. when a character logs on You should stop polling when the endpoint returns an empty dictionary, but have some fuzziness to make sure it wasn't just a quick disconnect.

(from https://developers.eveonline.com/blog/article/crest-updates-for-january-2016 under 'CHARACTER LOCATION')

raphendyr commented 8 years ago

We need to speed development on this area. On 11 Oct the IGB will be removed: https://community.eveonline.com/news/dev-blogs/bidding-farewell-to-the-in-game-browser/

I try to start working on it soon. I just need to allocate some time and setup test setup.

dagoaty commented 8 years ago

Let me know when you have something working and I'll get my test installation set up again.

proycon commented 8 years ago

Considering the announcement, this indeed needs to be a priority if the mapper is to survive, and it would be a shame if it didn't. I still prefer EWS over the other fancier-looking tools that have opened up recently. @raphendyr would be very welcome if you can indeed spearhead this .. I can help test.

dagoaty commented 8 years ago

I don't know if you have a rough idea of how you want to handle this. I'm thinking something like:

A new section under Settings->My Settings called something like 'Authenticated Characters'. This would have an add new character button which goes through the CREST authentication process. It would also list existing characters mapped against your account, plus the ability to delete them.
You need to be able to activate mapping for each character, so a "Pilots" drop down from the "baseNavDiv" navbar which contains a list of all CREST authenticated pilot names. Clicking on a name toggles the mapping for that character.
Finally, the polling code needs to handle an empty dictionary being returned causing mapping to be disabled. Presumably after (e.g.) 5-10 empty returns. It could be nice if this is notified to the logged in user owning that pilot, but not essential.

This is just ideas I'm throwing out. I wish I knew my way around Django enough to help code this.

raphendyr commented 8 years ago

Sounds good. I'm working with few django projects at work (you can notice from my profile) so that part will be easy, I just need to introduce my self to this project code.

About @dagoaty's points. Those sound good. Basically we need model to contain user with mappings to multiple EVE sso logins/crest tokens (so it doesn't matter what char you use to login after initial). Also tool needs to be able to merge 2 users (no need initially)

The menu to toggle tracking on and off is probably ok so you can disable tracking your characters. Though actual tracking I hope to be automatic. Keep it in 5-10s when player is online and drop to multiple minutes or so, when there is no response (se when he is online again, mapping is resumed).

As there might not be UI connected, I think silent mapping should be on. Maybe new connections should be marked as automatic or so. Need to iterate over this more.

We probably need to implement fleet interface at some point if the current ship was not available from other endpoints. Need to check that.

With ship info we can filter out pod kills/destructions. If new system and you are in pod then do not add new system.

Any ideas are really welcome here or in tweetfleet/#devfleet. I try to get my setup going on this week and maybe some initial work going on next.

dagoaty commented 8 years ago

The endpoint is cached for 10 seconds so there's little point going to a 5-10s poll. Realistically you're unlikely to cross multiple systems in 15-20 seconds so polling in that timescale would work.

CCP appear to want us to stop polling when we get an empty dictionary back. I like the idea of the mapper not needing intervention to start polling when we log in though. Maybe if a user is logged in and has a map open the polling should poke every 5 minutes until it gets a non-empty response then switch to the 15-20 second poll mode?

marbindrakon commented 8 years ago

I'll try to reactivate an account and work on this a little this weekend. This is the feature that made me realize EWS needed a ground-up rewrite that I don't have time for.

As a stop-gap, I will try implementing a structure stored in memcached with the current polling state and cache expiry of each character. Then a polling task running every 5s can scan that structure and only poll the ones whose cache timer is up and that have an active polling state.

The polling states would start as active when a user clicks a button in the UI to start tracking for a character and move to pending when an empty dictionary is recieved. Pending records would be polled every 30s for 5 minutes and then be removed (logoff assumed).

raphendyr commented 8 years ago

Work in progress version at https://github.com/raphendyr/eve-wspace/tree/feature/crest_wip if someone is interested to follow it. Comments are welcome in tweetfleet slack or github.

Got the eve sso example from https://github.com/flesser/django-crest-example integrated.

From here:

user needs map of characters associated to him (with CREST tokens)
implement way to get character objects (contain name and picture link etc)
groups need list of corporations that are part of it.
groups account has access to, should be loaded and cached to session in middleware. this cache should be invalidated when any change to group members are done.

raphendyr commented 7 years ago

I'm again working on this and I expect get this in some condition on this week.

EvilGrinUK commented 7 years ago

This is slightly more urgent now the IGB has been removed in the Ascension update (Nov 2016).

https://community.eveonline.com/news/dev-blogs/bidding-farewell-to-the-in-game-browser/

Without C.R.E.S.T. SSO and location support everything has to be inputted manually.

nyrocron commented 7 years ago

I hope to have time to work on this soon.

EvilGrinUK commented 7 years ago

That;'s great. I was worried this project was dead. If you need any help, let me know.

Maarten28 commented 7 years ago

In the branch "crest" you'll be able to find a working implemention of CREST. Current accounts are still used to login, however one account can hold multiple EVE account under the API button ("CREST Account Management).

In settings.py additional settings were added, which need to be updated before use:

CREST_ENABLED = False CREST_SECRET_KEY = 'xxxxxx' #to be created at https://developers.eveonline.com/ for TQ CREST_CLIENT_ID = 'xxxxxx' #to be created at https://developers.eveonline.com/ for TQ CREST_BASE_URL = 'Your URL here' #do not forget the / at the end and https if the connection is secure CREST_USER_AGENT = 'EVE W-space Instance'

For developers the scope is also included.

Currently the following items do need further development:

The location is pulled only once every 30 seconds for each char. It does not differentiate between online and offline chars, this can however be done and then scale it up.
Refresh tokens which do not work anymore (pulled every 20 minutes) are not being deleted from the database after x amount of failures.

Please test the branch, so it can be merged at a later date.

Maarten28 commented 7 years ago

Added ESI support and changed some naming around. It consists of 3 modules now:

SSO: For all auth related stuff
CREST support
ESI support: #261

The location is also pulled through ESI now, furthermore ship type and name were added again. ESI also has no rate limit, therefore the celery task runs every 15 seconds.

EvilGrinUK commented 7 years ago

If you follow the install docs and try and install the crest branch it'll break when you try and install the requirements in the virtualenv:

Collecting pycurl==7.43.0 (from -r /home/maptool/eve-wspace/requirements-mysql.txt (line 25))
  Downloading pycurl-7.43.0.tar.gz (182kB)
    100% |████████████████████████████████| 184kB 6.5MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-XHRWr7/pycurl/setup.py", line 823, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-build-XHRWr7/pycurl/setup.py", line 497, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-build-XHRWr7/pycurl/setup.py", line 71, in __init__
        self.configure()
      File "/tmp/pip-build-XHRWr7/pycurl/setup.py", line 107, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-XHRWr7/pycurl/

This can be fixed by running this command to install the missing packages on an Ubuntu 12.04 LTS server:

sudo apt-get install libcurl4-gnutls-dev librtmp-dev

Maarten28 commented 7 years ago

You're completely right, I'll update it later today.

This is what I get for never running a production environment :P.

EvilGrinUK commented 7 years ago

Also for creation of the Site API key it would be good to mention that you need to give the following scopes as it isn't obvious without looking in the code:

esi-location.read_location.v1: Allows reading of a character's active ship location esi-location.read_ship_type.v1: Allows reading of a character's active ship class

Another snag I've run into is that the SSO stuff is at the end of the settings.py That means that you can't override it in local_settings.py and actually enable SSO support. Rearranging things in settings.py fixes this.

The updated cron settings for celery were missing from local_settings.py so I ended up overriding the new ones with the old ones.

~~I'm also not getting the API menu showing up for normal users. I've not debugged as to why yet. It does show up as an admin though.~~ I was being dumb and hadn't given the correct permissions to the group.

Maarten28 commented 7 years ago

I've added libcurl4-gnutls-dev and librtmp-dev to the documentation apt-get command.
I've included the two scopes in a comment in the settings.py and local settings example file
I've pulled up the SSO/CREST/ESI settings in settings.py about the try/except
I've updated the example local settings with the latest celery schedule
I've removed the permission to view the API menu at all (there are still permissions in place for adding API keys)

EvilGrinUK commented 7 years ago

Great! Thanks for the hard work. I've updated my install and redone my local_settings.py. Now I'm at home I have tested with the game client running. I can add my CREST SSO login to the API fine however the map shows no indication of updating to include my presence or add systems when I jump. I think there something must be wrong with the polling. I'll dig into the celery jobs.

Maarten28 commented 7 years ago

You could run it through the python/django shell to determine whether the function is at least working properly.

from API.tasks import update_char_location
update_char_location()

If it then shows up on the mapper under pilots, at least you know with certainty it is the celery schedule/worker.

EvilGrinUK commented 7 years ago

I ran celeryd in debug mode. I got the following error:

[2016-12-20 19:22:26,463: ERROR/MainProcess] Task API.tasks.update_char_location[298ff8d2-e964-4966-8d43-48538bdd59bc] raised unexpected: NameError("global name 'esi_access_data' is not defined",)
Traceback (most recent call last):
  File "/home/maptool/eve-wspace/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/maptool/eve-wspace/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/maptool/eve-wspace/evewspace/API/tasks.py", line 49, in update_char_location
    response = esi_access_data(token,url)
NameError: global name 'esi_access_data' is not defined

Maarten28 commented 7 years ago

Fixed. :)

EvilGrinUK commented 7 years ago

And on to the next error:

[2016-12-20 19:27:31,787: ERROR/MainProcess] Task API.tasks.update_char_location[afb5b626-555d-4487-8080-f6c280ee5fb0] raised unexpected: NameError("global name 'Type' is not defined",)
Traceback (most recent call last):
  File "/home/maptool/eve-wspace/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/maptool/eve-wspace/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/maptool/eve-wspace/evewspace/API/tasks.py", line 60, in update_char_location
    current_ship = get_object_or_404(Type, pk=ship_type_id)
NameError: global name 'Type' is not defined

and

[2016-12-20 19:27:59,518: ERROR/MainProcess] Task API.tasks.update_char_location[7b479d99-9701-4239-a566-35d5638027a0] raised unexpected: KeyError('ship_type_id',)
Traceback (most recent call last):
  File "/home/maptool/eve-wspace/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/maptool/eve-wspace/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/maptool/eve-wspace/evewspace/API/tasks.py", line 56, in update_char_location
    ship_type_id = response2["ship_type_id"]
KeyError: 'ship_type_id'

and

[2016-12-20 19:28:28,721: ERROR/MainProcess] Task API.tasks.update_char_location[fc65efa9-b1c2-4d71-888f-c563c0de6fdd] raised unexpected: KeyError('solar_system_id',)
Traceback (most recent call last):
  File "/home/maptool/eve-wspace/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/maptool/eve-wspace/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/maptool/eve-wspace/evewspace/API/tasks.py", line 55, in update_char_location
    system_pk = response["solar_system_id"]
KeyError: 'solar_system_id'

Maarten28 commented 7 years ago

The first is solved. The second and third seem to be an issue on CCPs side, which I just checked on tweetfleet as well. The calls are throwing a 500 internal server error quite frequently with an uncaught exception. One thing I am currently looking into is how to do error handling, which are caused by CCP.

EvilGrinUK commented 7 years ago

I wondered what was going on! Thanks for your help tonight so far.

Presumably if the ESI call gets an error back then it'll just poll again?

Maarten28 commented 7 years ago

Since it seems that ESI is not that stable yet I've done it a bit differently. First, the system will throw a request at the ESI server twice, one for the location and one for the ship type/name. If the first (location) fails or gives no response a CREST request it submitted to gain that information through CREST. Ship type/name is considered an added benefit, if it returns empty the ship type/name are reported as "Unknown". This way we rely on the most modern method first, where CREST is the fallback. An additional scope has been added to the settings to facilitate this.

EvilGrinUK commented 7 years ago

Ok, thanks, that's working better in my root system now. It does occasionally find out the ship name + type (btw the labels are backwards in the pilot list and the system tooltip for these).

I tried jumping through an unmapped wormhole, It didn't add the new system to the map. I tried running the update_char_location() manually but I do not get any kind of stack trace this time. I intermittently get uncaught exception messages but I presume they are to do with the ESI server barfing HTTP 500 errors.

Maarten28 commented 7 years ago

Changed the ship name and type around, so it is correct now. Small fix 👍.

The jumping through an unmapped wormhole is indeed not implemented again, my concern was mostly making SSO/CREST/ESI work and show with one functionality how it was implemented.

I intermittently get uncaught exception messages but I presume they are to do with the ESI server barfing HTTP 500 errors.

I am unsure whether this is actually ESI, since I am handling this error (by creating a response of None). Did you update the scope of your SSO user to include the CREST endpoint (also on an application level on the dev website)? Otherwise this would throw exceptions.

EvilGrinUK commented 7 years ago

Thanks for that. Yes I did update. It wouldn't actually run until I cleared out the old SSO tokens from the database as I got 404 forbidden errors when it was requesting the CREST player location anyway.

When I run the code manually I get:

>>> update_char_location()
{u'error': u'uncaught exception'}

or

>>> update_char_location()
{u'error': u'uncaught exception'}
{u'error': u'uncaught exception'}

or it succeeds without any output.

Maarten28 commented 7 years ago

I know why, I put a print in to know how often it was going wrong. It is just the error handler being called :). I'll take it out.

EvilGrinUK commented 7 years ago

The jumping through an unmapped wormhole is indeed not implemented again

I'm guessing a rework of the code from _checkin_igb_trusted() in Map\views.py can be used to implement this?

Maarten28 commented 7 years ago

Probably, it is a bit more tricky since you are not sending a response to the client initiating the update (which you did in the IGB)... I have an idea how it could be done, but not sure it is a good idea.

EvilGrinUK commented 7 years ago

I'm guessing the CREST / ESI polls still run even if you aren't viewing the website. So presumably you'd have to detect if the player was logged into the web site too. If they were then pop up the add system dialog just like when manually adding systems (but with some info pre-populated). Otherwise silently add the system with some sensible defaults for links. I suspect getting this last bit to work is pretty straight forward.

Maarten28 commented 7 years ago

I'm guessing the CREST / ESI polls still run even if you aren't viewing the website.

Correct

So presumably you'd have to detect if the player was logged into the web site too. If they were then pop up the add system dialog just like when manually adding systems (but with some info pre-populated). Otherwise silently add the system with some sensible defaults for links. I suspect getting this last bit to work is pretty straight forward.

The first is the trickiest part, adding a system silently isn't a huge deal. The update char location already determines whether you are in a new system, thus allowing you to add it automatically following the logic which the IGB used.

I'm considering merging the crest branch in develop and create a new issue for the before mentioned items.

EvilGrinUK commented 7 years ago

Actually I though of a more elegant way to do it:

API always adds systems silently, as you said this is fairly straight forward.
Front end checks player system during a map refresh. If the system they are in is missing info or is new prompt for updates (based on a timestamp or a flag).

That way you don't ever need to worry if someone is logged in or not. It also means if other users can add information if the original system discoverer is logged off. Whoever gets to the system with the mapper open can be prompted to update it.

This could also built on to prompt for updates for sigs, wormhole status etc if the system data is determined to be 'stale'.

This all might need a few config options to let an admin make it more or less annoying for end users.

Maarten28 commented 7 years ago

The only use case which isn't covered is the ability to add blank systems. You press the + when scanning, it shows a black system on the map, while it also adds a system (silently). So that functionality needs to be disabled if we follow the logic above.

EvilGrinUK commented 7 years ago

That seems like a reasonable compromise to me?

Maarten28 commented 7 years ago

Just as a small fyi: I've removed libcurl/pycrest requirement, since a magic fairy told me that the requests library can achieve the same result.

EvilGrinUK commented 7 years ago

Alright. Can I switch from crest to the develop branch without breaking anything or would throwing the ec2 instance away and starting from scratch be better?

Maarten28 commented 7 years ago

The develop branch also includes a general change in requirements (version numbers) of the python modules. Your choice whether starting from scratch is easier.

Otherwise changing should not break anything, since pycrest/libcurl is more bloat than anything else :). I did refactor some code a moment ago, so maybe double check whether the develop branch is working as intended for you.

EvilGrinUK commented 7 years ago

I might try another Ubuntu 16.04 install and see if the updated requirements work on it actually.

Actually, I'll try both ways. Might as well be thorough.

evewspace / eve-wspace

[Feature] Crest API #162