Closed splatpope closed 5 months ago
Tkael
commented
1 year ago Agreed, API calls are counted independently per endpoint but the API becomes cripplingly rate limited if too many queries are performed.
Web scraping is generally considered discourteous to the week host and is not a viable solution here.
It may be possible to revise the search to use the Spansh API instead of EDSM.
splatpope
commented
1 year ago Agreed, API calls are counted independently per endpoint but the API becomes cripplingly rate limited if too many queries are performed.
Do you think there would be a way to get in touch with EDSM's staff so I could look into improving their API ? Being able to make more complicated queries corresponding to their website's capabilities would both enhance their product and reduce their eventual load, so I would suppose there would be demand for that.
Web scraping is generally considered discourteous to the week host and is not a viable solution here.
While I would agree in general terms, in this particular situation, it would be no different than having the user use their website to find out the nearest tech broker, which probably doesn't happen often enough to have a significant impact on load. On the other hand, retrieving system data for hundreds of systems does, which apparently warrants a hour long cooldown. I argue that it would actually benefit them for EDDI to use web crawling for this specific feature.
On the other hand, I do believe that it would indeed be graceful to work in agreement with EDSM's staff on such a service and make sure they aren't negatively impacted.
It may be possible to revise the search to use the Spansh API instead of EDSM.
I cannot find any resources pertaining to Spansh's API capabilities, can you direct me to them ? Also, I'll take advantage of that to state that Inara's API seems to be even more limited than EDSM's. Another solution would be to have EDDI host its own gateway to EDDN, but that seems somewhat out of scope.
All in all, the trend seems to be that the different mapping and searching services have limited APIs, even though they possess powerful web interfaces.
Tkael
commented
1 year ago EDSM has a Discord. I think it's linked from the EDSM website. But I don't think Anthor would be keen to have someone come in and rewrite their API (particularly because doing so could prove disruptive for developers currently using those resources).
The Spansh API is undocumented but Spansh freely encourages developers to use it anyways. If you open a developer console in your browser you can gather enough information to replicate the call and response in another application (we're already doing this for neutron star and carrier routing).
splatpope
commented
1 year ago EDSM has a Discord. I think it's linked from the EDSM website. But I don't think Anthor would be keen to have someone come in and rewrite their API (particularly because doing so could prove disruptive for developers currently using those resources).
Well, I would rather help release a new version, which fits with the fact that the current API is explicitly called V1. I suppose they wouldn't just stop servicing it. I see no net downsides, I'll just try to contact the relevant people on that discord and see.
The Spansh API is undocumented but Spansh freely encourages developers to use it anyways. If you open a developer console in your browser you can gather enough information to replicate the call and response in another application (we're already doing this for neutron star and carrier routing).
This seems like an interesting solution indeed, I'll dig around and see what can be done.
Tkael
commented
5 months ago Closing as a duplicate of #2327.
What's Wrong (please be as specific as possible)
Plotting a ship route for station services in the Navigation Monitor tab can be exceedingly slow and inefficient, leading to some instances in which the search doesn't conclude in a timely manner (we're talking upwards of 10 minutes without a response, and we will see below that this delay is most likely to be one hour long), mostly when searching for tech brokers. For the other services, searches tend to resolve successfully, but sometimes after more than 20 seconds of unresponsiveness.
The fact that there are no timeouts on those searches, or that they simply cannot be cancelled can be very damageable to user experience regarding this specific feature. These are separate issues that are arguably even more critical.
This particular issue seems to stem from a cooldown issue with EDSM.
I will rather describe below a situation in which I'm in the Nu Pictoris system, looking for the nearest Guardian Tech Broker. According to EDSM, which to my understanding is the source of the data used by these searches, the proper route should be a direct jump to the Latuba system, then supercruise to the Soukup City station. This intended target is well within the advertised limits of the search feature.
Expected
A two waypoint route in the relevant list view, the first of which is the current system "Nu Pictoris", the second of which being the "Latuba" system, "Soukup City" station.
Observed
The search does not conclude and is stuck, as evidenced by the "Search" button displaying a looping animation, and the text-to-speech program warning me that a route search is in progress when trying to setup a new, easier search (i.e. nearest fuel star).
The latter symptom can not appear when using EDDI directly, as the search interface does not allow for plotting a new route while a query is in progress; thus, this appeared when using EDDI with VoiceAttack, but note that it can be safely concluded that VA isn't implicated in the issue.
Steps to reproduce
While in the Nu Pictoris system, ask the "Plot Ship Route" feature of the "Navigation Monitor" tab to find the nearest Guardian or Human Technology Broker. To truly reproduce my particular experience, search for all other services than tech brokers beforehand.
Configuration
My Investigation
Investigation Notes
One thing I did was to read the verbose logs. The one attached contains output for multiple successful searches originating from the Nu Pictoris system. The last one is the guardian tech broker search mentioned above.
One can easily see that the search gets a list of systems from EDSM, then for each and all of them, it gets the list of its celestial bodies from EDSM in separate requests then parses the contents. This seems consistent with the capabilities of EDSM's API.
Some time during this process, EDSM complains about too many requests and the search seems to be stuck for the duration of EDSM's cooldown, i.e. one hour (!!!).
It seems that all the relevant code for this feature is in NavigationService/QueryResolvers/ServiceQueryResolvers.cs
I do see that query results are supposed to be cached to a SQLite database, which seems to explain why the other searches seemed to become much faster after the first time doing them. (Although this first time is still exceedingly long). One can probably deduce that since tech broker searches have a much larger range, it is normal that more queries have to be made than for the other types of searches, these queries being logically not cached yet.
The aforementioned database cache is probably why the log doesn't show similar output for the previous successful searches, as I had performed them multiple times before during testing, in a previous instance of EDDI.
Considering all of the above, if the amount of EDSM requests generated by the searches is actually reasonible, then the problem might be with EDSM itself and its absurdly long cooldown. (to be honest, I'm 50/50 on this)
EDDI Logs
[Any log files providing additional details about what happened. These are located at %APPDATA%/EDDI and have the extension ".log"] eddi.log
Player journals
[These are not required if EDDI Logs are verbose but may be helpful otherwise. They are located at [your Saved Games folder]/Frontier Developments/Elite Dangerous/. Please only provide excerpts or attach files from the play session where the issue occurred.]
All of this notwithstanding, I do not understand why a full search has to be performed in a cube around the player to find specific services.
On EDSM's website, it is possible to search for specific services, simultaneously sorting the results by distance to commander (by supplying the current commander position). Sadly, it seems that their API does not support crafting such a query.
One could directly use EDSM's (or inara's for that matter) station service search directly from their site, using web crawling. Obviously, this solution is highly sensitive to their website's layout.