[wishlist] Iterable thrift structures

[manukranthk]

Currently the way we can have iterable thrift structures is by having a List of items. We can access the items of the list only after we have the entire list. Also, we need to de-serialize all the elements in order to read even the very first element. This might add unnecessary memory pressure on the system.

[allComputableThings]

It looks straightforward to build a codec to do this. However, as far as vanilla thrift RPC spec goes, isn't this a limitation of its design?

By default, I think that the auto-generated client code won't result a result until the entire 'thing' has been received because the thrift type language doesn't know about lists. To do what you want you could either. It might be worth contact the thrift mailing list:

• are there proposals to add iterators to the thrift type language?
• what workaround to people employ:

I'd be interested in how people workaround this limitation. It seems possible, for example to have a API return an object that represents an iterator reference and then make additional API calls to get more items:

TIterator<TUser> queryDB();
List<User> getMore( TIterator<TUser>, int pageSize );

where

class TIterator{ int iteratorID; }

and the sever maintains a map:

activeIterators Map<int, Iterator

[andrewcox]

Yeah, the way thrift currently works, you can't get what you want (to start iterating through results before the whole result arrives).

However, we are considering potentially adding streaming support to fbthrift, for exactly this kind of thing, after which point we'd add it to swift as well.

People who need this now here at FB, do pretty much what you're saying: they break the result up into chunks, and make multiple calls to get it.

[ghost]

Thank you for reporting this issue and appreciate your patience. We've notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.