Split up xmlprotocol.scala

[devinus]

Unfortunately, I have to start by saying that much to my chagrin I cannot give an example of the WSDL and accompanying XSD files that generates the 65KLOC xmlprotocol.scala, as it's proprietary and I'm bound by non-disclosure agreements between my company and the provider.

However, I can give as much detail and my own observations:

The file I'm sure is clean Scala code and would probably eventually compile given enough time and memory made available to the sbt process. However, my machine only has 4g. At first I was receiving OutOfMemory exceptions during the compilation process on that file that forced me to use -Xmx2g to make more heap space available, then I was receiving errors about the process not having enough memory per stack, so I increased the stack size to -Xss1m as well. Still, the file would simply take too long to compile.

Also, a quick search turns up that the file contains about 3k instances of the implicit keyword.

I was just wondering if there was a way to split up the XMLProtocol trait into logically smaller pieces. Is there any way to split a trait up amongst multiple files?

[eed3si9n]

Others have reported similar issue with larger size code (like #131), and if the problem is coming from implicit resolution, splitting them up ultimately doesn't change the situation. One possibility may be to stop the internal use of implicits by explicitly passing the protocols to the argument. This would reduce the flexibility, but would at least work, if it works.

[devinus]

How much work would it be to change it to explicit resolution? I'm here to use and test any experimental branch if you ever set it up.

[eed3si9n]

Not sure, but I'll try hacking around soon.

[devinus]

Okay. I'll be ready to test it. I hate making it less flexible, but at least we'll be able to see if it is indeed all the implicit resolution.

[eed3si9n]

It's implemented in try/explicit branch. I have a wsdl that stackoverflows normally, and it's stackoverflowing..

[devinus]

@eed3si9n As in, it's still stackoverflowing?

[eed3si9n]

Yea. So if you want you can give it a shot, by publishing it locally, but don't keeps your hope too high.

[devinus]

I'm giving it a go right now...

[devinus]

Unfortunately, I don't think it was implicit resolution anymore. I'm almost positive it's just that the Scala compiler cannot handle a 65KLOC file. I'm not sure any compiler handles a 65KLOC file that well.

Perhaps, and just brainstorming, but could the XMLProtocol trait be split amongst files? E.g.

trait A {}
trait B {}
trait C {}
trait ABC extends A with B with C {}

[eed3si9n]

ok. I'll keep this open as a potential enhancement in some future major release.

[hoff2]

I'm exploring this, as I have a couple of ~80kloc xmlprotocol.scala files I can't compile, but it seems that about 80% of them (line-wise) is all the *TypeFormat classes, which are each quite small on their own, it's just that there are 4000+ of them. Putting all those in their own trait/file probably wouldn't help -- that file would still be 78kloc or so :(

[adamdecaf]

So I found https://github.com/eed3si9n/scalaxb/commit/32ae67d9a6509eb9f8eca44ef908fdabcfd3dd49 from @eed3si9n and merged it into current master. There's just one test that's failing that I'm trying to fix before I'd PR it against scalaxb master, but if anyone wants to help that'd be awesome. Here's the merge so far https://github.com/Banno/scalaxb/pull/2

[jypma]

If anyone is still on this, a good use case to bring scalaxb to its knees is the UBL 2.2 specification (open, freely downloadable from here, there's an xsd folder in the zip file). 60K LoC xmlprotocol.scala. It does eventually compile with -Xmx8G:

[success] Total time: 434 s (07:14), completed Mar 6, 2020 8:53:21 AM

on a Thinkpad T540p ...

The same XSDs give XJC no problem at all. Then again, no implicits over there ;-)

[mirelon]

This issue affects also my usecase:

Multiple WSDL files for different 3rd party APIs. Trying to separate the generated classes into multiple packages. It works, but: xmlprotocol.scala is present in one of those packages and contains classes and methods from all those APIs! And defaultScope contains all the namespaces in one NamespaceBinding.

What if there are two APIs with a conflict in a class name? We need to separate xmlprotocol.scala into packages and it is not possible because of this issue.