iron-io / functions

IronFunctions - the serverless microservices platform by
https://iron.io
Apache License 2.0
3.18k stars 227 forks source link

Optionally store async output? #173

Open treeder opened 8 years ago

treeder commented 8 years ago

Sync responds with stdout output and logs stderr.

Async logs both stdout and stderr. Perhaps we could optionally store the async output temporarily to support workflows and piping together functions down the road.

When creating route, user could decide:

"output": "logs|store"
ucirello commented 8 years ago

I feel half-hearted about this idea. For sure, it would be very handy to have this feature implemented in Functions, thus facilitating one very common use case. On the other hand, it opens a Pandora Box of hard decisions:

1 - Where should we store the output? I am tempted to say that it shouldn't be in the datastore. Perhaps add a support for services like S3 and Minio? If using Minio, should we embed it as we do with BoltDB?

2 - Some design pain points: a) How long should the each result be stored? b) Supposing results may expire, where do we set their expiration time? When the code is added or when the code is executed? c) Should we add an API extension to manipulate these stored results? (Of course, the user could just operate the storage service, still he would have a hard time telling stale from useful results). d) Is this API extension part of a future Open Functions Standard? e) Is this orthogonal to a future workflow support? Is this orthogonal to Functions itself? (In the sense this could be developed as a long running function instead of a built-in feature?)

I think it is a good idea and we should debate it more thoroughly.

ucirello commented 8 years ago

This could relate to https://github.com/iron-io/functions/pull/200

denismakogon commented 7 years ago

@ccirello I believe that IronFunctions can integrate with Logstash for storing logs because S3 is cloud-locking, so no way to support, let's say, OpenStack Swift, Mimio is not designed to work with logs specifically rather than with files/objects.