Closed istreeter closed 4 years ago
I had mixed feelings about expanding the list of applications. Here are some of the thoughts going round in my head:
It might be relevant that there are schemas for the configs, btw (more of an FYI): https://github.com/snowplow/iglu-central/tree/master/schemas/com.snowplowanalytics.dataflowrunner
I share both of your doubt - not knowing why it was there in the first place leaves me in doubt as to whether we're missing something.
Re: supporting other applications, there's also the factor that including them implies that they're tested and supported, and there's a maintenance overhead to that. If the demand to support another application is there, I think I'd rather encourage that people raise issues and pull requests than accidentally create an expectation. However I don't feel strongly about this, it's a vague gut feeling more than anything.
When I consider these two points you've just made:
there's also the factor that including them implies that they're tested and supported
Maybe we should abandon the list of allowed applications altogether. If the command line contains an invalid application then aws will respond with an error message.
I lean towards removing the check altogether. By listing them, I think it suggests they are supported but by removing the check then we are making no promises in the code that we support all these options (and README/guides would only include instructions for those we know work and are tested). (still wonder why it was added in the first place though...)
@chuwy have you got an opinion on the above conversation?
I agree with Paul and Ian on "removing the check altogether" and wouldn't be too concerned about why we added this list initially. Or I rather have too many ideas why we added it and only one of them (documentation purposes) is worth to consider.
Also, I really doubt we ever used/tested Pig, Mahout or even Hive with Dataflow Runner.
I removed the check of valid application names. This is now the error would receive if you run dataflow-runner up
with an invalid application name:
ERRO[0000] ValidationException: Specified application: Snowplow is invalid.
status code: 400, request id: bdb66ae0-30a2-4b2a-b464-77b641dcfaa4
ValidationException: Specified application: Snowplow is invalid.
status code: 400, request id: bdb66ae0-30a2-4b2a-b464-77b641dcfaa4
The exit code is 1.
This change was required to let us run flink applications on EMR using dataflow-runner.