Add public transit to Susi AI

[Fluepke]

Many public transit providers have open data portals offering realtime data. (E.g. for Berlin see this). We should access this data directly or via a wrapper service to provide Susi users information about different means of transportation Maybe the sourcecode from transportr could be used to achieve this, especially enabler

[Fluepke]

@mariobehling suggested to implement this skill via dream.asksusi.com This would require to have an external API, that can provide the necessary data. Another idea would be to implement this via a ConsoleService, but it might be problematic to include and distribute the necessary API keys.

[mariobehling]

@Fluepke Let's do the lightweight approach first. "dream.asksusi.com" is not intended to be a way for implementation. The idea is to provide a setting to write skills and test them right away. Currently skills need to be added to the repository. We are planning to provide an easier way, but this is out of scope of this issue.

We really need developers to write skills and understand the flow. So, for the moment using external APIs is the best approach and it moves the project further a lot. Please proceed.

[sudheesh001]

@Fluepke This definitely sounds like an interesting thing to do. The intention with susi is that others who might want to use this data can also integrate it. We can always use a whitelisted IP Address of the server to ensure that the API can be used between susi and the required service by distributing and installing a key on the servers but the intention is to work with open data.

Providing API ingestion interfaces is definitely something that needs to be implemented into susi server but for starters we can always implement this as a susi dream or a consoleService and use the conf to store the API credentials or we could as well build an API manager.

[Orbiter]

This issue shows that one of the core requirements for a public storage place must be a concept to hide application keys. I added that idea to https://github.com/fossasia/susi_server/issues/106#issuecomment-294828643

[Fluepke]

Ok, so I've tinkered around with enabler. It's working quite well and for Berlin / Brandenburg it does not require any API keys as it just parses a public available interface (is this legal)? Whats the way to go for implementing a ConsoleService for this? Just copy the necessary code from enabler (while contributing the authors) and writing a ConsoleService to interface with enabler?

Another option would be to deploy enabler with all API keys to a private server. One could access this server from Susi. But I'd prefer a solution that does not depend from somebody running a service on his server.

What do you think?

[Fluepke]

We can include transport-enabler as a git submodule and add it to the build chain. However there were some arguments against git submodules. What's your opinion towards this? If it's ok, I would write a proper PR for this (not that hacky one from a hackathon)

[sudheesh001]

It's working quite well and for Berlin / Brandenburg it does not require any API keys as it just parses a public available interface (is this legal)?

As long as it's published online for webscrapers and crawlers to index it, it's fine storing it and doing anything we want with it.

I don't really like the idea of having git-submodules in the repository, it causes a huge dependency and lots of problems for new comers. For now I would really like the second approach to have the enabler on a private server and someone running it as a service from there. We can then replicate and host the enabler ourselves on the production servers.

[Fluepke]

Of course, we could host transportr-enabler, however hosting stuff on our own requires us to keep it all running and to maintain it.

Downsides of hosting transportr-enabler:

• Maintenance, availability
• Newcomers wanting to work on public transit would need to setup a server locally
• Need for a 2nd project (for the server software)
  • Thus both, susi_server and that server software would need to be synchronised (there's a dependency that is not being enforced)
  • Testing across two projects is hard
  • It becomes complex if there is more than one server to rely on
• The server would also require a git-submodule
  • We really should reuse available (well written) code, so reimplementing public transit APIs is no way to go for
• We would need to define an API for susi_server to communicate to this server software, which is again commicating to an API. xkcd
• Legal issues may apply when redistributing information from some API (see also T&C's)

On the other, submodules definitely come at a cost as well:

• Project size increases drastically, thus more error potential, longer compile times
• Maybe dependency issues, when API changes. On the other hand, the impact of an external API implementation should be rather small to the core project.