Filter payments by asset

[bartekn]

Some organizations (like exchanges) need to stream payments connected to a single asset. This way they have a granular control to disable a specific asset and make sure that cross deposits are not possible. This looks like a very important feature for exchanges.

[MonsieurNicolas]

A few questions:

1. payments or effects (trades in particular)?
2. we already have endpoints that allow people to stream, isn't client side filtering easy to implement?
   • Maybe we should start with that?
   • Server side seems to imply that we're going to keep adding more and more complex filters to APIs, and it's unclear to me where we would draw the line.
3. if the reason for this is to reduce operating cost, it might be better to solve it with a combination of selective ingestion (storage is a huge cost driver) and client side filtering?

[bartekn]

payments or effects (trades in particular)?

I think both actually.

we already have endpoints that allow people to stream, isn't client side filtering easy to implement?

This was my first response when talking with exchanges. Usually exchanges have a separate infrastructure set for each asset. Say you want to support XLM and MOBI but for any reason you want to suspend MOBI deposits. Then, in case of a single stream, when you turn it back on you need to check the history again, skipping all non-MOBI deposits (sometimes thousands).

Server side seems to imply that we're going to keep adding more and more complex filters to APIs, and it's unclear to me where we would draw the line.

Good point. GraphQL (@nullstyle actually had some plans to implement this in Horizon) could solve this for us but I'm not sure about performance implications. However, we're still in Rest API world so I'm thinking that:

• In short term we should probably keep adding features that are really useful for a majority of our community (exchanges is a big part of it).
• In the long term we should allow more customizations in Horizon by adding a plugin system. Something that requires a much longer discussion and design.

if the reason for this is to reduce operating cost, it might be better to solve it with a combination of selective ingestion (storage is a huge cost driver) and client side filtering?

Again, this also requires a longer discussion what's the best way to solve it. Currently you can't do it without modifying Horizon code.

[MonsieurNicolas]

say you want to support XLM and MOBI but for any reason you want to suspend MOBI deposits. Then, in case of a single stream, when you turn it back on you need to check the history again, skipping all non-MOBI deposits (sometimes thousands).

Oh I see, so the scenario is to temporary put a hold on processing certain payments. I don't think this is a real problem: yes it's a bit inefficient to have to reprocess historical data, but it seems to be a once a year type of thing and client side filtering works well enough for this scenario.

re: GraphQL I agree with you, I don't think getting GraphQL APIs will just be a "drop in", instead we'll have to make deeper rearchitecture of Horizon in order to hit scalability goals.

As far as investments server side, I would rather focus on allowing people to write custom filters server side: this is where we're going to see a real explosion of new interesting features from the community. Is the issue that right now our Horizon data model is too complicated or something else?

Reason I am a bit opposed to more complexity in current Horizon is that it's more technical debt if we can't tease out the various layers: I think of the "non filtering" APIs (most of current APIs), as the base layer of Horizon's data model, any processing done on the data (filtering for example), should be done on top of that base layer. Only with a layered approach we're going to be able to continue to efficiently invest in the overall system.