Integration tests (sbt scripted) fail for the current master branch (0.2.0)

[zyv]

Here is the abridged output of sbt scripted, I suspect that the failing test have to do with the output tagged as [error]:

[error] [pool-11-thread-5] WARN de.zalando.swagger.ValidationsConverter$ - Ignoring nonsense validations for date: maxLength(10), minLength(0), pattern("""/[1-9][A-Z0-9]*/""".r)
[error] [pool-11-thread-5] WARN de.zalando.swagger.ValidationsConverter$ - Ignoring nonsense validations for date-time: maxLength(10), minLength(1), pattern("""/[1-9][A-Z0-9]*/""".r)
[error] [pool-11-thread-5] WARN de.zalando.swagger.ValidationsConverter$ - Ignoring nonsense validations for date-time: maxLength(10), minLength(0), pattern("""/[1-9][A-Z0-9]*/""".r)
[error] [pool-11-thread-5] WARN de.zalando.swagger.ValidationsConverter$ - Ignoring nonsense validations for date: maxLength(10), minLength(0), pattern("""/[1-9][A-Z0-9]*/""".r)
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path(), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint GET /pets/{id} , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path("schema" / "model"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path("schema" / "model"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path("schema" / "model"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint GET /pets/findByTags , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint PUT /pets , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint POST /pets , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint GET /pets/{petId} , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint POST /pets/{petId} , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint DELETE /pets/{petId} , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint GET /pets/findByStatus , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("stores" / "order"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users" / "createWithList"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call PUT Path("users" / "{username}"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users" / "createWithArray"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("stores" / "order"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users" / "createWithList"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call PUT Path("users" / "{username}"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users" / "createWithArray"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("string"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("string2"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("string"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("string2"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path(), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path(), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path("test-path" / "{id}"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint GET /pets/findByTags , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint PUT /pets , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint POST /pets , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint GET /pets/{petId} , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint POST /pets/{petId} , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint DELETE /pets/{petId} , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] INFO de.zalando.apifirst.generators.ScalaPlayCallEnricher - Not generating tests for secure API endpoint GET /pets/findByStatus , this issue will be addressed in future versions of the plugin
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("stores" / "order"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users" / "createWithList"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call PUT Path("users" / "{username}"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users" / "createWithArray"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("stores" / "order"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users" / "createWithList"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call PUT Path("users" / "{username}"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path("users" / "createWithArray"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path("schema" / "model"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path("schema" / "model"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path("schema" / "model"), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call GET Path(), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path(), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path(), defaulting to MimeType("application/json")
[error] [pool-11-thread-3] WARN de.zalando.apifirst.generators.ScalaPlaySpecEnricher - No mime-types is defined for call POST Path(), defaulting to MimeType("application/json")
...
[info] [error] Failed: Total 98, Failed 59, Errors 0, Passed 39
[info] [error] Failed tests:
[info] [error]  numbers_validation.yaml.Numbers_validationYamlSpec
[info] [error]  heroku.petstore.api.yaml.HerokuPetstoreApiYamlSpec
[info] [error]  nakadi.yaml.NakadiYamlSpec
[info] [error]  cross_spec_references.yaml.Cross_spec_referencesYamlSpec
[info] [error]  full.petstore.api.yaml.FullPetstoreApiYamlSpec
[info] [error]  echo.EchoApiYamlSpec
[info] [error]  form_data.yaml.Form_dataYamlSpec
[info] [error]  error_in_array.yaml.Error_in_arrayYamlSpec
[info] [error]  admin.SimplePetstoreApiYamlSpec
[info] [error]  hackweek.yaml.HackweekYamlSpec
[info] [error]  split.petstore.api.yaml.SplitPetstoreApiYamlSpec
[info] [error]  string_formats_validation.yaml.String_formats_validationYamlSpec
[info] [error]  uber.api.yaml.UberApiYamlSpec
[info] [error] (test:test) sbt.TestsFailedException: Tests unsuccessful
[info] [error] Total time: 57 s, completed Dec 19, 2016 12:00:06 PM
[info] + swagger / compile
[success] Total time: 333 s, completed Dec 19, 2016 12:00:07 PM

[slavaschmidt]

@zyv Hey, thank you for the issue.

The failing tests are by design. The idea is that at the moment you provide a specification, the tests and controller skeletons are generated. But at this moment the tests should fail because there is no implementation in controllers yet. That's exactly what we want to see in our scripted tests - after initial artefacts were generated, the tests are red.

Does it make sense?

[zyv]

Hi Slava,

Thank you for the explanation! I can roughly follow what you are saying, but I would argue that it's very confusing to see lots of failures as a result of running scripted tests, so maybe the output could be somehow rearranged to make it more clear what's going on & what's expected and/or document this. However, I can't propose a better solution, because I'm not knowledgeable enough about your code; so feel free to close this issue...

--Yury.

[slavaschmidt]

@zyv Hey, thanks for your feedback. One idea could be instead of returning "Not implemented" from freshly generated controllers to return some random generated data. This would make tests pass, but the downside is that the developer cannot distinguish anymore between tests passing because of the proper implementation or just because of this random data returned from the API.

Leaving this issue open to investigate the possibilities further.

[zyv]

Yes, returning random data doesn't sound like a great option at all... Moreover, there is always a chance that the tests will accidentally pass.

I would have actually expected that expected failures are interpreted as success and unexpected successes as failures in the integration tests context, only as I said I have no constructive suggestions on how to implement that.

[slavaschmidt]

That's exactly what is happening during the scripted test, I guess log entry prefix [info] [error] should give a hint that this is a nested test configuration in which outer test expects inner test to fail.