ValeLang / Vale

Compiler for the Vale programming language - http://vale.dev/
https://vale.dev/
Apache License 2.0
1.79k stars 54 forks source link

Unicode capability #125

Open Ivo-Balbaert opened 3 years ago

Ivo-Balbaert commented 3 years ago

Vale is not yet Unicode capable. Here is a test program:

fn main() export {
  english = "Hello world from Vale!"; 
  println(english);

  // Unicode:
  espanol := "¡Hola, mundo!";    
  lao := "ສະ​ບາຍ​ດີ​ຊາວ​ໂລກ";
  urdu := "!ہیلو ، دنیا";
  emoji := "👋🌍❗";
  greek_greeting = "Καλημέρα κόσμε"; 
  asian_greeting = "こんにちは 世界";

  println(espanol);
  println(lao);
  println(urdu);
  println(emoji);
}

which gives as output:

C:\Vale\code_examples>python ..\valec.py build hello_world_international.vale Using Visual Studio at C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC. Running: java -cp C:\Vale\Valestrom.jar net.verdagon.vale.driver.Driver build -o build.vast hello_world_international.vale Internal error while compiling [PureWindowsPath('hello_world_international.vale')]:

Exception in thread "main" java.nio.charset.UnmappableCharacterException: Input length = 1 at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:275) at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:352) at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:188) at java.base/java.io.InputStreamReader.read(InputStreamReader.java:181) at java.base/java.io.BufferedReader.fill(BufferedReader.java:161) at java.base/java.io.BufferedReader.readLine(BufferedReader.java:326) at java.base/java.io.BufferedReader.readLine(BufferedReader.java:392) at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:70) at scala.collection.Iterator.foreach(Iterator.scala:944) at scala.collection.Iterator.foreach$(Iterator.scala:944) at scala.collection.AbstractIterator.foreach(Iterator.scala:1432) at scala.collection.TraversableOnce.addString(TraversableOnce.scala:357) at scala.collection.TraversableOnce.addString$(TraversableOnce.scala:353) at scala.collection.AbstractIterator.addString(Iterator.scala:1432) at scala.collection.TraversableOnce.mkString(TraversableOnce.scala:323) at scala.collection.TraversableOnce.mkString$(TraversableOnce.scala:322) at scala.collection.AbstractIterator.mkString(Iterator.scala:1432) at scala.collection.TraversableOnce.mkString(TraversableOnce.scala:325) at scala.collection.TraversableOnce.mkString$(TraversableOnce.scala:325) at scala.collection.AbstractIterator.mkString(Iterator.scala:1432) at net.verdagon.vale.driver.Driver$.readCode(Driver.scala:82) at net.verdagon.vale.driver.Driver$.$anonfun$build$1(Driver.scala:89) at scala.collection.immutable.List.map(List.scala:283) at net.verdagon.vale.driver.Driver$.build(Driver.scala:89) at net.verdagon.vale.driver.Driver$.buildAndOutput(Driver.scala:179) at net.verdagon.vale.driver.Driver$.main(Driver.scala:259) at net.verdagon.vale.driver.Driver.main(Driver.scala)

MoxyFoxy commented 3 years ago

I believe Odin uses utf8proc, which is a great library for this, though I don't know the difficult of using C code in Scala. I feel that, if possible, it would be a nice addition to the compiler.