search

clulab / eidos

Machine reading system for World Modelers
Apache License 2.0
36 stars 24 forks source link

Deduplication error during sbt assembly #1098

Closed bgyori closed 2 years ago

bgyori commented 2 years ago

I haven't tried doing this for a while but I found that there are issues with sbt assembly again on the latest master:

[error] 2 errors were encountered during merge
[error] java.lang.RuntimeException: deduplicate: different file contents found in the following:
[error] /Users/ben/Library/Caches/Coursier/v1/http/artifactory.cs.arizona.edu%3A8081/artifactory/sbt-release/org/clulab/processors-models/0.1.10/processors-models-0.1.10.jar:LICENSE
[error] /Users/ben/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.14/scala-library-2.12.14.jar:LICENSE
[error] /Users/ben/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.12.14/scala-reflect-2.12.14.jar:LICENSE
[error] deduplicate: different file contents found in the following:
[error] /Users/ben/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/apple/AppleJavaExtensions/1.4/AppleJavaExtensions-1.4.jar:META-INF/INDEX.LIST
[error] /Users/ben/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/objenesis/objenesis/3.2/objenesis-3.2.jar:META-INF/INDEX.LIST
[error]     at sbtassembly.Assembly$.applyStrategies(Assembly.scala:161)
[error]     at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:43)
[error]     at sbtassembly.Assembly$.x$1$1(Assembly.scala:41)
[error]     at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:41)
[error]     at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:41)
[error]     at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:86)
[error]     at sbtassembly.Assembly$.inputs$1(Assembly.scala:76)
[error]     at sbtassembly.Assembly$.apply(Assembly.scala:103)
[error]     at sbtassembly.Assembly$.$anonfun$assemblyTask$1(Assembly.scala:278)
[error]     at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error]     at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62)
[error]     at sbt.std.Transform$$anon$4.work(Transform.scala:68)
[error]     at sbt.Execute.$anonfun$submit$2(Execute.scala:282)
[error]     at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23)
[error]     at sbt.Execute.work(Execute.scala:291)
[error]     at sbt.Execute.$anonfun$submit$1(Execute.scala:282)
[error]     at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error]     at sbt.CompletionService$$anon$2.call(CompletionService.scala:64)
[error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[error]     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[error]     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]     at java.lang.Thread.run(Thread.java:748)
[error] (assembly) deduplicate: different file contents found in the following:
[error] /Users/ben/Library/Caches/Coursier/v1/http/artifactory.cs.arizona.edu%3A8081/artifactory/sbt-release/org/clulab/processors-models/0.1.10/processors-models-0.1.10.jar:LICENSE
[error] /Users/ben/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.12.14/scala-library-2.12.14.jar:LICENSE
[error] /Users/ben/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.12.14/scala-reflect-2.12.14.jar:LICENSE
[error] deduplicate: different file contents found in the following:
[error] /Users/ben/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/apple/AppleJavaExtensions/1.4/AppleJavaExtensions-1.4.jar:META-INF/INDEX.LIST
[error] /Users/ben/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/objenesis/objenesis/3.2/objenesis-3.2.jar:META-INF/INDEX.LIST

I tried deleting various caches but keep getting the same.

MihaiSurdeanu commented 2 years ago

Thanks @bgyori !

I am not sure what is causing this. Maybe @kwalcock can take a look?

kwalcock commented 2 years ago

I did see this come in yesterday. It shouldn't take long to figure out. Probably a duplicate file was added to an updated library dependency and needs to be filtered. It will hopefully be taken care of by Monday.

MihaiSurdeanu commented 2 years ago

Thanks Keith!

On Sat, Nov 27, 2021 at 12:33 Keith Alcock @.***> wrote:

I did see this come in yesterday. It shouldn't take long to figure out. Probably a duplicate file was added to an updated library dependency and needs to be filtered. It will hopefully be taken care of by Monday.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/clulab/eidos/issues/1098#issuecomment-980789814, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAI75TRPTYTCRMOADR7OIMLUOEXBHANCNFSM5I3U5HHA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

kwalcock commented 2 years ago

This is being addressed with #1099, which is being tested right now.

kwalcock commented 2 years ago

The tests passed, so I merged the branch and will close this. The tests don't try to run assembly, but it was working on the local computer. What wasn't tested at all was the result of the assembly. I hope @bgyori will report any problems with that, whether related to this issue or anything else that has happened since last time.

bgyori commented 2 years ago

Thanks a lot, I'm trying it now! This has happened a few times in the past (#1098, #823, #794, #694), I wonder if - despite it being a resource intensive and somewhat slow task - it would be possible to test this (i.e., sbt assembly) automatically so that we can see if there are issues with it. What do you think?

kwalcock commented 2 years ago

Maybe there's some subset of assembly that can be more quickly tested that includes the merging but not actually the slow building of the final jar. I'll see if there is an obvious way to do that.

kwalcock commented 2 years ago

The complete assembly just took 25 minutes on my computer (in power saving mode but with an SSD that the server doesn't have), so I don't think it's an option. I don't see any easy way to run only part of the assembly. The messages about merge strategies being applied appears fairly quickly, but I don't see an easy way to stop processing there. I could perhaps fork the assembly plugin and have it stop early or maybe use some script that runs sbt assembly and observes the log for a short while before killing the process. It's not obvious what to do.

We will probably be publishing eidos to maven more often soon. One could then have a completely different project that uses the published artifact and runs assembly from there. This would not work on just any master branch of eidos unless perhaps one published it locally. Otherwise, it is like using an older version of eidos.