search

com-lihaoyi / mill

Mill is a fast JVM build tool that supports Java and Scala. 2-4x faster than Gradle and 4-10x faster than Maven for common workflows, Mill aims to make your project’s build process performant, maintainable, and flexible
https://mill-build.org/
MIT License
2.2k stars 353 forks source link

CharsetProviders defined by dependencies aren't available while running tests #895

Closed alec-bernardi closed 2 months ago

alec-bernardi commented 4 years ago

When using libraries that supply a custom CharsetProvider, the provided Charset implementations are not available in tests.

I am not super knowledgeable with regard to classloading in the JVM, but it seems this stems from the fact that the CharsetProvider implementations are loaded with the system classloader, while mill's TestRunner loads the runtime classpath of the test using the context classloader of a subprocess. Since the JVM isn't started with JCharset on the classpath, I don't believe the ServiceLoader will be able to find the META-INF/services/java.nio.charset.spi.CharsetProvider within jcharset-2.1.jar.

This issue is demonstrated on OpenJDK 8 by the following example project, which depends on https://www.freeutils.net/source/jcharset/ for the ISO-8859-8-BIDI character encoding:

./build.sc:

import mill._
import mill.scalalib._

object foo extends ScalaModule {
  def scalaVersion = "2.12.10"
  def ivyDeps = Agg(ivy"net.freeutils:jcharset:2.1")

  object test extends Tests {
    def ivyDeps = Agg(ivy"org.scalatest::scalatest:3.0.8")
    def testFrameworks = Seq("org.scalatest.tools.Framework")
  }
}

./foo/test/src/CharsetTest.scala:

import org.scalatest._
import java.nio.charset.Charset

class CharsetTest extends FunSpec with Matchers with Inside {
  describe("The set of available Charsets") {
    it("should contain ISO-8859-8-BIDI") {
      Charset.availableCharsets() should contain key "ISO-8859-8-BIDI"
    }
  }
}

The output of mill foo.test:

[55/55] foo.test.test 
CharsetTest:
The set of available Charsets
- should contain ISO-8859-8-BIDI *** FAILED ***
  {"Big5"=Big5, "Big5-HKSCS"=Big5-HKSCS, "CESU-8"=CESU-8, "EUC-JP"=EUC-JP, "EUC-KR"=EUC-KR, "GB18030"=GB18030, "GB2312"=GB2312, "GBK"=GBK, "IBM-Thai"=IBM-Thai, "IBM00858"=IBM00858, "IBM01140"=IBM01140, "IBM01141"=IBM01141, "IBM01142"=IBM01142, "IBM01143"=IBM01143, "IBM01144"=IBM01144, "IBM01145"=IBM01145, "IBM01146"=IBM01146, "IBM01147"=IBM01147, "IBM01148"=IBM01148, "IBM01149"=IBM01149, "IBM037"=IBM037, "IBM1026"=IBM1026, "IBM1047"=IBM1047, "IBM273"=IBM273, "IBM277"=IBM277, "IBM278"=IBM278, "IBM280"=IBM280, "IBM284"=IBM284, "IBM285"=IBM285, "IBM290"=IBM290, "IBM297"=IBM297, "IBM420"=IBM420, "IBM424"=IBM424, "IBM437"=IBM437, "IBM500"=IBM500, "IBM775"=IBM775, "IBM850"=IBM850, "IBM852"=IBM852, "IBM855"=IBM855, "IBM857"=IBM857, "IBM860"=IBM860, "IBM861"=IBM861, "IBM862"=IBM862, "IBM863"=IBM863, "IBM864"=IBM864, "IBM865"=IBM865, "IBM866"=IBM866, "IBM868"=IBM868, "IBM869"=IBM869, "IBM870"=IBM870, "IBM871"=IBM871, "IBM918"=IBM918, "ISO-2022-CN"=ISO-2022-CN, "ISO-2022-JP"=ISO-2022-JP, "ISO-2022-JP-2"=ISO-2022-JP-2, "ISO-2022-KR"=ISO-2022-KR, "ISO-8859-1"=ISO-8859-1, "ISO-8859-13"=ISO-8859-13, "ISO-8859-15"=ISO-8859-15, "ISO-8859-2"=ISO-8859-2, "ISO-8859-3"=ISO-8859-3, "ISO-8859-4"=ISO-8859-4, "ISO-8859-5"=ISO-8859-5, "ISO-8859-6"=ISO-8859-6, "ISO-8859-7"=ISO-8859-7, "ISO-8859-8"=ISO-8859-8, "ISO-8859-9"=ISO-8859-9, "JIS_X0201"=JIS_X0201, "JIS_X0212-1990"=JIS_X0212-1990, "KOI8-R"=KOI8-R, "KOI8-U"=KOI8-U, "Shift_JIS"=Shift_JIS, "TIS-620"=TIS-620, "US-ASCII"=US-ASCII, "UTF-16"=UTF-16, "UTF-16BE"=UTF-16BE, "UTF-16LE"=UTF-16LE, "UTF-32"=UTF-32, "UTF-32BE"=UTF-32BE, "UTF-32LE"=UTF-32LE, "UTF-8"=UTF-8, "windows-1250"=windows-1250, "windows-1251"=windows-1251, "windows-1252"=windows-1252, "windows-1253"=windows-1253, "windows-1254"=windows-1254, "windows-1255"=windows-1255, "windows-1256"=windows-1256, "windows-1257"=windows-1257, "windows-1258"=windows-1258, "windows-31j"=windows-31j, "x-Big5-HKSCS-2001"=x-Big5-HKSCS-2001, "x-Big5-Solaris"=x-Big5-Solaris, "x-COMPOUND_TEXT"=x-COMPOUND_TEXT, "x-euc-jp-linux"=x-euc-jp-linux, "x-EUC-TW"=x-EUC-TW, "x-eucJP-Open"=x-eucJP-Open, "x-IBM1006"=x-IBM1006, "x-IBM1025"=x-IBM1025, "x-IBM1046"=x-IBM1046, "x-IBM1097"=x-IBM1097, "x-IBM1098"=x-IBM1098, "x-IBM1112"=x-IBM1112, "x-IBM1122"=x-IBM1122, "x-IBM1123"=x-IBM1123, "x-IBM1124"=x-IBM1124, "x-IBM1166"=x-IBM1166, "x-IBM1364"=x-IBM1364, "x-IBM1381"=x-IBM1381, "x-IBM1383"=x-IBM1383, "x-IBM300"=x-IBM300, "x-IBM33722"=x-IBM33722, "x-IBM737"=x-IBM737, "x-IBM833"=x-IBM833, "x-IBM834"=x-IBM834, "x-IBM856"=x-IBM856, "x-IBM874"=x-IBM874, "x-IBM875"=x-IBM875, "x-IBM921"=x-IBM921, "x-IBM922"=x-IBM922, "x-IBM930"=x-IBM930, "x-IBM933"=x-IBM933, "x-IBM935"=x-IBM935, "x-IBM937"=x-IBM937, "x-IBM939"=x-IBM939, "x-IBM942"=x-IBM942, "x-IBM942C"=x-IBM942C, "x-IBM943"=x-IBM943, "x-IBM943C"=x-IBM943C, "x-IBM948"=x-IBM948, "x-IBM949"=x-IBM949, "x-IBM949C"=x-IBM949C, "x-IBM950"=x-IBM950, "x-IBM964"=x-IBM964, "x-IBM970"=x-IBM970, "x-ISCII91"=x-ISCII91, "x-ISO-2022-CN-CNS"=x-ISO-2022-CN-CNS, "x-ISO-2022-CN-GB"=x-ISO-2022-CN-GB, "x-iso-8859-11"=x-iso-8859-11, "x-JIS0208"=x-JIS0208, "x-JISAutoDetect"=x-JISAutoDetect, "x-Johab"=x-Johab, "x-MacArabic"=x-MacArabic, "x-MacCentralEurope"=x-MacCentralEurope, "x-MacCroatian"=x-MacCroatian, "x-MacCyrillic"=x-MacCyrillic, "x-MacDingbat"=x-MacDingbat, "x-MacGreek"=x-MacGreek, "x-MacHebrew"=x-MacHebrew, "x-MacIceland"=x-MacIceland, "x-MacRoman"=x-MacRoman, "x-MacRomania"=x-MacRomania, "x-MacSymbol"=x-MacSymbol, "x-MacThai"=x-MacThai, "x-MacTurkish"=x-MacTurkish, "x-MacUkraine"=x-MacUkraine, "x-MS932_0213"=x-MS932_0213, "x-MS950-HKSCS"=x-MS950-HKSCS, "x-MS950-HKSCS-XP"=x-MS950-HKSCS-XP, "x-mswin-936"=x-mswin-936, "x-PCK"=x-PCK, "x-SJIS_0213"=x-SJIS_0213, "x-UTF-16LE-BOM"=x-UTF-16LE-BOM, "X-UTF-32BE-BOM"=X-UTF-32BE-BOM, "X-UTF-32LE-BOM"=X-UTF-32LE-BOM, "x-windows-50220"=x-windows-50220, "x-windows-50221"=x-windows-50221, "x-windows-874"=x-windows-874, "x-windows-949"=x-windows-949, "x-windows-950"=x-windows-950, "x-windows-iso2022jp"=x-windows-iso2022jp} did not contain key "ISO-8859-8-BIDI" (CharsetTest.scala:7)
1 targets failed
foo.test.test CharsetTest The set of available Charsets should contain ISO-8859-8-BIDI
alec-bernardi commented 4 years ago

I'm willing to bet this is related to #716

lefou commented 1 year ago

Is this still valid? Meanwhile, we reworked our TestRunner a bit, so chances are high, this issue is already fixed.

lefou commented 1 year ago

Also, can you point to a working example?

lihaoyi commented 2 months ago

Going to call this fixed since we reworked the testrunner classloader hierarchy in https://github.com/com-lihaoyi/mill/commit/f944ceb96565385953ba7fe6869100da24025b03 which makes the user code run in the root classloader