Signature serialization is different on Windows

jonahbeckford commented 5 months ago

When I use the following from an opam switch lib directory:

"$PROJECT_DIR/codept_bin/codept" -o "$PROJECT_DIR/src/Common/RuntimeSig.ml.temp" -k -sig \
    angstrom/*.cmi \
    ...

I get the following on Linux and macOS:

((version(0 11 0))(namespace((Angstrom(M((origin(Unit((Local(angstrom angstrom.cmi))(Angstrom))))...

but this on Windows:

((version(0 11 0))(namespace((Angstrom(M((origin(Unit((Local(angstrom.cmi))(Angstrom))))...

It doesn't matter which package is used.

Any ideas where it could be breaking down? I can do print line debugging if I had an idea where to start in the code. (Usually when I see something like this, it is because / is hardcoded as the directory separator somewhere in the code. But I don't see any obvious culprits.)

Octachron commented 5 months ago

The fact that the directory is elided is definitively suspicious. What does the atlas part of the -deps option looks like in this case?

jonahbeckford commented 5 months ago

Windows Experiment 1

I'm using MSYS2 on Windows to make the command line comparison easier:

/y/source/dksdk-coder/.ci/o/dkml/lib $ /y/source/dksdk-coder/codept_bin/codept.exe -k -deps angstrom/*.cmi
{
"version" : [0, 11, 0],
"dependencies" :
  [{ "file" : "angstrom/angstrom.cmi" },
  {
  "file" : "angstrom/angstrom__.cmi",
  },
  { "module" : ["Angstrom__Input"], "mli" : "angstrom/angstrom__Input.cmi" },
  { "module" : ["Angstrom__More"], "mli" : "angstrom/angstrom__More.cmi" },
  {
  "module" : ["Angstrom__Parser"],
  "mli" : "angstrom/angstrom__Parser.cmi"
  }]
}

/y/source/dksdk-coder/.ci/o/dkml/lib $ ls angstrom/*.cmi
angstrom/angstrom.cmi    angstrom/angstrom__Buffering.cmi       angstrom/angstrom__Input.cmi  angstrom/angstrom__Parser.cmi
angstrom/angstrom__.cmi  angstrom/angstrom__Exported_state.cmi  angstrom/angstrom__More.cmi

jonahbeckford commented 5 months ago

macOS Experiment 1

lib % ../../../../codept_bin/codept -k -deps angstrom/*.cmi
{
"version" : [0, 11, 0],
"dependencies" :
  [{ "file" : "angstrom/angstrom.cmi" },
  {
  "file" : "angstrom/angstrom__.cmi",
  "deps" :
    [["Angstrom__Parser"], ["Angstrom__More"], ["Angstrom__Input"],
    ["Angstrom__Exported_state"], ["Angstrom__Buffering"]]
  }, { "file" : "angstrom/angstrom__Buffering.cmi" },
  { "file" : "angstrom/angstrom__Exported_state.cmi" },
  { "file" : "angstrom/angstrom__Input.cmi" },
  { "file" : "angstrom/angstrom__More.cmi" },
  { "file" : "angstrom/angstrom__Parser.cmi" }],
"local" :
  [{ "module" : ["Angstrom"], "mli" : "angstrom/angstrom.cmi" },
  { "module" : ["Angstrom__"], "mli" : "angstrom/angstrom__.cmi" },
  {
  "module" : ["Angstrom__Buffering"],
  "mli" : "angstrom/angstrom__Buffering.cmi"
  },
  {
  "module" : ["Angstrom__Exported_state"],
  "mli" : "angstrom/angstrom__Exported_state.cmi"
  },
  { "module" : ["Angstrom__Input"], "mli" : "angstrom/angstrom__Input.cmi" },
  { "module" : ["Angstrom__More"], "mli" : "angstrom/angstrom__More.cmi" },
  {
  "module" : ["Angstrom__Parser"],
  "mli" : "angstrom/angstrom__Parser.cmi"
  }]
}
lib % ls angstrom/*.cmi
angstrom/angstrom.cmi                   angstrom/angstrom__Buffering.cmi        angstrom/angstrom__Input.cmi            angstrom/angstrom__Parser.cmi
angstrom/angstrom__.cmi                 angstrom/angstrom__Exported_state.cmi   angstrom/angstrom__More.cmi

jonahbeckford commented 5 months ago

Windows Experiment 2

Err ... apparently there is some non-determinism (perhaps caching or .cmi recompilation) occurring.

When I run the command for the second time on Windows (nothing else was done!) it outputs the complete set of .cmi:

/y/source/dksdk-coder/.ci/o/dkml/lib $ /y/source/dksdk-coder/codept_bin/codept.exe -deps angstrom/*.cmi
{
"version" : [0, 11, 0],
"dependencies" :
  [{ "file" : "angstrom/angstrom.cmi" },
  {
  "file" : "angstrom/angstrom__.cmi",
  "deps" :
    [["Angstrom__Parser"], ["Angstrom__More"], ["Angstrom__Input"],
    ["Angstrom__Exported_state"], ["Angstrom__Buffering"]]
  }, { "file" : "angstrom/angstrom__Buffering.cmi" },
  { "file" : "angstrom/angstrom__Exported_state.cmi" },
  { "file" : "angstrom/angstrom__Input.cmi" },
  { "file" : "angstrom/angstrom__More.cmi" },
  { "file" : "angstrom/angstrom__Parser.cmi" }],
"local" :
  [{ "module" : ["Angstrom"], "mli" : "angstrom/angstrom.cmi" },
  { "module" : ["Angstrom__"], "mli" : "angstrom/angstrom__.cmi" },
  {
  "module" : ["Angstrom__Buffering"],
  "mli" : "angstrom/angstrom__Buffering.cmi"
  },
  {
  "module" : ["Angstrom__Exported_state"],
  "mli" : "angstrom/angstrom__Exported_state.cmi"
  },
  { "module" : ["Angstrom__Input"], "mli" : "angstrom/angstrom__Input.cmi" },
  { "module" : ["Angstrom__More"], "mli" : "angstrom/angstrom__More.cmi" },
  {
  "module" : ["Angstrom__Parser"],
  "mli" : "angstrom/angstrom__Parser.cmi"
  }]
}

/y/source/dksdk-coder/.ci/o/dkml/lib $ /y/source/dksdk-coder/codept_bin/codept.exe -k -deps angstrom/*.cmi
{
"version" : [0, 11, 0],
"dependencies" :
  [{ "file" : "angstrom/angstrom.cmi" },
  {
  "file" : "angstrom/angstrom__.cmi",
  "deps" :
    [["Angstrom__Parser"], ["Angstrom__More"], ["Angstrom__Input"],
    ["Angstrom__Exported_state"], ["Angstrom__Buffering"]]
  }, { "file" : "angstrom/angstrom__Buffering.cmi" },
  { "file" : "angstrom/angstrom__Exported_state.cmi" },
  { "file" : "angstrom/angstrom__Input.cmi" },
  { "file" : "angstrom/angstrom__More.cmi" },
  { "file" : "angstrom/angstrom__Parser.cmi" }],
"local" :
  [{ "module" : ["Angstrom"], "mli" : "angstrom/angstrom.cmi" },
  { "module" : ["Angstrom__"], "mli" : "angstrom/angstrom__.cmi" },
  {
  "module" : ["Angstrom__Buffering"],
  "mli" : "angstrom/angstrom__Buffering.cmi"
  },
  {
  "module" : ["Angstrom__Exported_state"],
  "mli" : "angstrom/angstrom__Exported_state.cmi"
  },
  { "module" : ["Angstrom__Input"], "mli" : "angstrom/angstrom__Input.cmi" },
  { "module" : ["Angstrom__More"], "mli" : "angstrom/angstrom__More.cmi" },
  {
  "module" : ["Angstrom__Parser"],
  "mli" : "angstrom/angstrom__Parser.cmi"
  }]
}

jonahbeckford commented 5 months ago

Hypothesis 1

Since I'm passing in Unix-style paths (ex. angstrom/angstrom__Input.cmi) the following code is misbehaving:

https://github.com/Octachron/codept/blob/38e0ffb5cde6c982e7a471dca6a6ec1cfa3d137d/lib/namespaced.ml#L98-L105

On Unix that would be:

let nms = []
let name = "Angstrom__Input"
let _filename :: r = "angstrom__Input.cmi" :: ["angstrom"]
let namespace = ["angstrom"]

On Windows that would be:

let nms = []
let name = "Angstrom__Input"
let _filename :: r = "angstrom/angstrom__Input.cmi" :: []
let namespace = []

I'll try to pass Windows paths to codept and see if that is a fix.

EDIT 1: Confirmed the hypothesis on Windows using Format.eprintf statements.

Octachron / codept