Config files without EOF newline in dependencies causes parsing errors

Naillik1 commented 3 years ago

Hi! Came across this problem with config file concatenation, and it seems like a bug -- if it's the intended behaviour, I'd love to know where I went wrong.

Problem

I have two Scala projects: app & lib, where lib is included as a dependency of app. Both app & lib depend on typesafe config 1.4.0 via PureConfig 0.13.0. Each has a reference.conf.

Recently, an end-of-file newline was removed from lib/../reference.conf, causing various parsing errors upon attempting to load configs in app. Which particular error is thrown depends on the contents of the first line of app/.../reference.conf.

Examples

Base

// app/.../Main.scala
import com.typesafe.config.ConfigFactory
object Main extends App {
    ConfigFactory.load()
}

The following examples are the results of varying either lib/.../reference.conf or app/.../reference.conf, then re-building and running app (sbt assembly; java -jar /out/app/assembly/dest/out.jar)

1. Primitive on first line of app/.../reference.conf yields `ConfigException`: `Parse`

# lib/.../reference.conf 
setting = snag # <EOF, no newline>

# app/.../reference.conf
foo = bar

com.typesafe.config.ConfigException$Parse:
reference.conf @ jar:file:/out/app/assembly/dest/out.jar!/reference.conf: 2713: 
Expecting end of input or a comma, got '=' (if you intended '=' to be part of a key or string value, try enclosing the key or value in double quotes, or you may be able to rename the file .properties rather than .conf)

Debugging this was entirely opaque due to the error message referring to the line-number in the concatenated file set; essentially a matter of trial-and-error removing configs until the issue could be isolated to the first line of the file.

2. Object on first line of app/.../reference.conf yields `ConfigException`: `WrongType`

# lib/.../reference.conf 
setting = snag # <EOF, no newline>

# app/.../reference.conf
foo {
  bar = "baz"
}

com.typesafe.config.ConfigException$WrongType: 
reference.conf @ jar:file:/out/app/assembly/dest/out.jar!/reference.conf: 2713:
Cannot concatenate object or list with a non-object-or-list, Unquoted("snagfoo") and SimpleConfigObject({"bar":"baz"}) are not compatible

I was lucky enough to both own the offending lib code and to recognize that snag was a config value found in lib. Without both, I'd be no closer to knowing what's going on.

3. Inserting a newline at first line of app/.../reference.conf properly loads config

# lib/.../reference.conf 
setting = snag # <EOF, no newline>

# app/.../reference.conf

foo = bar

4. Inserting a newline at lib/.../reference.conf end-of-file properly loads config

# lib/.../reference.conf
setting = snag

# app/.../reference.conf
foo = bar

I'd expect config file concatenation to insert a break between files to guard against this; is there a reason why concatenating config files doesn't do this? Or is there some other mechanism I'm missing that I should be doing?

If it's indeed a bug, I'm happy to contribute a fix (though I'd appreciate any pointers as to where to start).

havocp commented 3 years ago

I don't think the config library itself does textual concatenation- it may be the problem here is in a fat jar build plugin that concats files textually? maybe it could add a newline?

raboof commented 3 years ago

So what you're seeing is the reference.conf files get concatenated without inserting newlines.

This is not something typesafe-config can help: the files are concatenated by your build tool - in this case, by the sbt-assembly plugin.

I remember seeing the behavior you describe (and making sure my reference.conf files have a newline at the end). However, looking at the sources for the 'concat' merge strategy of sbt-assembly, it seems it should be inserting a newline if there is none: https://github.com/sbt/sbt-assembly/blob/v0.15.0/src/main/scala/sbtassembly/MergeStrategy.scala#L66 . That code seems to have been there for quite a while, too.

If you expect your library to be used in projects that are not using sbt, it might be smart to just make sure the newline is there - even if you fix sbt-assembly, other build tools/plugins might be harder to configure to insert those newlines.

If it's indeed a bug, I'm happy to contribute a fix (though I'd appreciate any pointers as to where to start).

This looks like it might be a bug in sbt-assembly, so I'd start looking closer there.

Naillik1 commented 3 years ago

@raboof Ah, that looks to be it -- I've been using mill, and taking a closer look concatenation during mill's assembly task appears to be the issue. Thanks for pointing me in the right direction!

lightbend / config