Circle TMPDIR issue

[gaku-sei]

I first noticed this issue while using the tailwind ppx: https://github.com/dylanirlbeck/tailwind-ppx/issues/92, but it seems to be related to BuckleScript (since I already faced this issue before using the the above ppx, just not as often).

To summarize the issue, BS seems to fail ~80% of the time (sometimes more often) on circle ci. The exceptions are totally random, and can be raised by any modules. Most of the time though, the exception is End_of_file, but it's sometimes different 😞

Also, notice that it's getting worse and worse now our project is bigger (~1000 modules).

Facing the issue with BS 7.3.2

Here is the bsconfig used, if that helps:

{
  "name": "app",
  "reason": {
    "react-jsx": 3
  },
  "sources": [
    {
      "dir": "src",
      "subdirs": true
    },
    {
      "dir": "tests",
      "subdirs": true,
      "type": "dev"
    }
  ],
  "package-specs": [
    {
      "module": "es6",
      "in-source": true
    }
  ],
  "suffix": ".bs.js",
  "namespace": true,
  "bs-dependencies": [
    "bs-fetch",
    "reason-react",
    "decco",
    "bs-express",
    "reason-urql",
    "bs-bastet",
    "relude",
    "relude-parse",
    "bs-optic",
    "bs-webapi",
    "wonka"
  ],
  "bs-dev-dependencies": ["@glennsl/bs-jest"],
  "ppx-flags": [
    ["@dylanirlbeck/tailwind-ppx/tailwind-ppx", "-path ./assets/styles.css"],
    ["@baransu/graphql_ppx_re/ppx6", "-lean-parse"],
    "decco/ppx",
    "bs-optic/lenses-ppx"
  ],
  "refmt": 3,
  "warnings": {
    "number": "-48",
    "error": "+A+8+11+12+16+26+27+31+32+33+34+35+39+44+45+101+102+104+105"
  },
  "bsc-flags": ["-bs-super-errors", "-open Relude_Globals"],
  "gentypeconfig": {
    "language": "typescript",
    "importPath": "relative",
    "shims": {
      "ReactEvent": "ReactEvent"
    },
    "debug": {
      "all": false,
      "basic": false
    }
  }
}

[bobzhang]

is CircleCI log available?

[gaku-sei]

Sure 😄

Here is part of the result of the bsb -make-world command:

$ bsb -make-world 
[1/8] Building src/bs_fetch.mlast
[2/8] Building src/Fetch.mliast 
[3/8] Building src/Fetch.mlast
[4/8] Building src/bs_fetch.d 
[5/8] Building src/Fetch.d
[6/8] Building src/Fetch.cmi 
[7/8] Building src/Fetch.cmj
[8/8] Building src/bs_fetch.cmj

// ... Goes on ...

FAILED: src/components/atoms/Menu.reast  
/home/circleci/workspace/node_modules/bs-platform/linux/bsc.exe  -w -48 -warn-error +A+8+11+12+16+26+27+31+32+33+34+35+39+44+45+101+102+104+105 -color always -bs-jsx 3 -ppx '/home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css' -ppx '/home/circleci/workspace/node_modules/@baransu/graphql_ppx_re/ppx6 -lean-parse' -ppx /home/circleci/workspace/node_modules/decco/ppx -ppx /home/circleci/workspace/node_modules/bs-optic/lenses-ppx -bs-super-errors -open Relude_Globals -o src/components/atoms/Menu.reast -bs-syntax-only -bs-binary-ast /home/circleci/workspace/packages/app/src/components/atoms/Menu.re

Fatal error: exception End_of_file

  We've found a bug for you!

  /home/circleci/workspace/packages/app/src/components/atoms/Menu.re

  Error while running external preprocessor

Command line: /home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css '/tmp/camlppxae4f71' '/tmp/camlppxc06f32'

 // Goes on a bit ...

FAILED: src/components/atoms/Footer.reast 
 /home/circleci/workspace/node_modules/bs-platform/linux/bsc.exe  -w -48 -warn-error +A+8+11+12+16+26+27+31+32+33+34+35+39+44+45+101+102+104+105 -color always -bs-jsx 3 -ppx '/home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css' -ppx '/home/circleci/workspace/node_modules/@baransu/graphql_ppx_re/ppx6 -lean-parse' -ppx /home/circleci/workspace/node_modules/decco/ppx -ppx /home/circleci/workspace/node_modules/bs-optic/lenses-ppx -bs-super-errors -open Relude_Globals -o src/components/atoms/Footer.reast -bs-syntax-only -bs-binary-ast /home/circleci/workspace/packages/app/src/components/atoms/Footer.re

Fatal error: exception End_of_file

  We've found a bug for you!

  /home/circleci/workspace/packages/app/src/components/atoms/Footer.re

  Error while running external preprocessor

Command line: /home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css '/tmp/camlppx7a0e14' '/tmp/camlppxef3e37'

 [268/1023] Building src/components/atoms/Button.reast
 FAILED: subcommand failed.

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

lerna ERR! yarn run bootstrap stderr:
error Command failed with exit code 2. 
error Command failed with exit code 2.

lerna ERR! yarn run bootstrap exited 2 in 'app'
error Command failed with exit code 2.

info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Exited with code exit status 2 

So here is the failing command:

/home/circleci/workspace/node_modules/bs-platform/linux/bsc.exe  -w -48 -warn-error +A+8+11+12+16+26+27+31+32+33+34+35+39+44+45+101+102+104+105 -color always -bs-jsx 3 -ppx '/home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css' -ppx '/home/circleci/workspace/node_modules/@baransu/graphql_ppx_re/ppx6 -lean-parse' -ppx /home/circleci/workspace/node_modules/decco/ppx -ppx /home/circleci/workspace/node_modules/bs-optic/lenses-ppx -bs-super-errors -open Relude_Globals -o src/components/atoms/Menu.reast -bs-syntax-only -bs-binary-ast /home/circleci/workspace/packages/app/src/components/atoms/[Can be any component].re

For more background: the command is ran using yarn through lerna

[bobzhang]

Error while running external preprocessor by reading the log, it is indeed coming from ppx

[bobzhang]

CircleCI may have some issues with temp directory: https://stackoverflow.com/questions/32856071/spec-creating-tmp-file-on-circleci-is-failing

[bobzhang]

can you ssh into CircleCI and run the failing command one by one. Note the failed command is purely parsing, it is a relatively simple task

[gaku-sei]

oh, that's really good to know, thank you 😄

Let me try using ssh 👍

So a good solution could be to force the creation of a tmp folder on Circle ci?

[gaku-sei]

So when running the same commands via ssh, it works perfectly 🤔

The /tmp seems to be emptied a bit too early when the taks is ran by Circle ci though

[gaku-sei]

~Weirdly enough, it seems that mkdir -p /tmp at the beginning of the job solves this issue 😕 Not sure why, since that folder already exists, and is properly populated some times ^^~

My bad, it worked only a few times.

What's even weirder, is that, when running one of the failing command (for instance:

/home/circleci/workspace/node_modules/bs-platform/linux/bsc.exe  -w -48 -warn-error +A+8+11+12+16+26+27+31+32+33+34+35+39+44+45+101+102+104+105 -color always -bs-jsx 3 -ppx '/home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css' -ppx '/home/circleci/workspace/node_modules/@baransu/graphql_ppx_re/ppx6 -lean-parse' -ppx /home/circleci/workspace/node_modules/decco/ppx -ppx /home/circleci/workspace/node_modules/bs-optic/lenses-ppx -bs-super-errors -open Relude_Globals -o src/components/molecules/input/PasswordInput.reast -bs-syntax-only -bs-binary-ast /home/circleci/workspace/packages/app/src/components/molecules/input/PasswordInput.re

) it doesn't raise any exception.

But running the ppx alone:

/home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css '/tmp/camlppx798f09' '/tmp/camlppx32de69'

Will fail (since the files "disappeared").

I also tried to force an other tmp folder, but got the same issues 😞

[bobzhang]

@gaku-sei thanks for digging, it seems this indeed comes from temp files issue, will check if there is a more robust solution. Can you set TMPDIR to a specific path?

[gaku-sei]

Thank you @bobzhang

I tried to set the TMPDIR to an other dir, and create the dir, but faced the exact same issues 😞

[bobzhang]

Hi, we will do some more logging around ppx and get back to you later

On Fri, Jun 19, 2020 at 3:16 PM Kévin Combriat notifications@github.com wrote:

Thank you @bobzhang https://github.com/bobzhang

I tried to set the TMPDIR to an other dir, and create the dir, but faced the exact same issues 😞

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/BuckleScript/bucklescript/issues/4469#issuecomment-646479692, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFWMKYZ7P5D5DBRTLTEY6DRXMGF5ANCNFSM4OBL47GQ .

-- Regards -- Hongbo Zhang

[bobzhang]

@gaku-sei the current ppx handling will remove those temp files right after it is read. I sent a [PR] (https://github.com/BuckleScript/bucklescript/pull/4475) that it will only delete files after all ppx succeed. Also you can try bsc -verbose ... to log those ppx commands. Is is easy for you to try it to see more verbose diagnostics from CircleCI

[gaku-sei]

Wouah! That's awesome, thank you @bobzhang Let me try this out as soon as possible 😄

[gaku-sei]

While trying BS 8.1.0-dev1, I tried to debug a bit more, but still no luck 😞

Given the following error:

[50/1023] Building src/components/pages/ContactUs.reast 
FAILED: src/components/pages/ContactUs.reast 

/home/circleci/workspace/node_modules/bs-platform/linux/bsc.exe  -w -48 -warn-error +A+8+11+12+16+26+27+31+32+33+34+35+39+44+45+101+102+104+105 -color always -bs-jsx 3 -ppx '/home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css' -ppx '/home/circleci/workspace/node_modules/@baransu/graphql_ppx_re/ppx6 -lean-parse' -ppx /home/circleci/workspace/node_modules/decco/ppx -ppx /home/circleci/workspace/node_modules/bs-optic/lenses-ppx -bs-super-errors -open Relude_Globals -o src/components/pages/ContactUs.reast -bs-syntax-only -bs-binary-ast /home/circleci/workspace/packages/app/src/components/pages/ContactUs.re

Fatal error: exception End_of_file

  We've found a bug for you!

  /home/circleci/workspace/packages/app/src/components/pages/ContactUs.re

  Error while running external preprocessor

Command line: /home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css '/tmp/ppx95fcd3ContactUs.re' '/tmp/ppx18e94dContactUs.re'

I tried to run the first bsc command manually and using the verbose flag:

/home/circleci/workspace/node_modules/bs-platform/linux/bsc.exe -verbose -w -48 -warn-error +A+8+11+12+16+26+27+31+32+33+34+35+39+44+45+101+102+104+105 -color always -bs-jsx 3 -ppx '/home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css' -ppx '/home/circleci/workspace/node_modules/@baransu/graphql_ppx_re/ppx6 -lean-parse' -ppx /home/circleci/workspace/node_modules/decco/ppx -ppx /home/circleci/workspace/node_modules/bs-optic/lenses-ppx -bs-super-errors -open Relude_Globals -o src/components/pages/ContactUs.reast -bs-syntax-only -bs-binary-ast /home/circleci/workspace/packages/app/src/components/pages/ContactUs.re

+ /home/circleci/workspace/node_modules/bs-platform/linux/refmt.exe --print=binary /home/circleci/workspace/packages/app/src/components/pages/ContactUs.re > /tmp/bspp24ab45

+ /home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css '/tmp/ppxd73ab2ContactUs.re' '/tmp/ppx34bb54ContactUs.re'

+ /home/circleci/workspace/node_modules/@baransu/graphql_ppx_re/ppx6 -lean-parse '/tmp/ppx34bb54ContactUs.re' '/tmp/ppxfef778ContactUs.re'

+ /home/circleci/workspace/node_modules/decco/ppx '/tmp/ppxfef778ContactUs.re' '/tmp/ppxa55135ContactUs.re'

+ /home/circleci/workspace/node_modules/bs-optic/lenses-ppx '/tmp/ppxa55135ContactUs.re' '/tmp/ppx94dd40ContactUs.re'

and it "works".

The second command:

/home/circleci/workspace/node_modules/@dylanirlbeck/tailwind-ppx/tailwind-ppx -path ./assets/styles.css '/tmp/ppx95fcd3ContactUs.re' '/tmp/ppx18e94dContactUs.re'

works as well when ran manually.

The /tmp/ppx95fcd3ContactUs.re file contains some binaries, and the second one /tmp/ppx18e94dContactUs.re is empty before running the ppx command again, and properly populated after.

So it seems the command fails randomly, even though the files are present, and when ran manually, they just work as expected.

Also, I got the same error locally once, so it seems it's not only on Circle ci 😞

[bobzhang]

@gaku-sei could it be a bug coming from tailwind-ppx? On bsc side, we are doing very simple things, literally just call the command -- what would happen when multiple tailwind-ppx is spawned simultaneously. Edit: the chance is not low that the cause is from tailwind, since all the logs pointing that something goes wrong when runing tailwind

[gaku-sei]

I agree, it looks like it unfortunately 😕 I created an issue on the repo here https://github.com/dylanirlbeck/tailwind-ppx/issues/92, it might be worth digging a bit what the ppx does indeed 😕

Thank you 😄

[bobzhang]

@gaku-sei I will close this issue since unclear what I can do here. But let me know if you want more logging to help diagnose

[bobzhang]

@gaku-sei I got some insight from other users.

Hi Bob, I've been digging into the CircleCi Failed Marshalled Build issue from a couple weeks ago and I may have found a bug ninja uses sysconf(_SC_NPROCESSORS_ONLN) to detect the amount of parallelism and creates a pool (unless you use -j N right) ubuntu docker containers will report the # of CPUs on the system. CircleCI runs on 36+ core machines so ninja spawns 36 processes to build as many branches as it can at the same time i think because the graphql_ppx doesn't use a spinlock, it corrupts the process and leads to a race condition which breaks the build so we can fix this in our setup by explicitly setting the -j flag like this: ./node_modules/bs-platform/linux/bsb.exe -make-world -- -j 5 our builds take ~5-10x longer now but we don't have any issues. I'm not sure if this makes sense or not but wanted to let you know either way, thank you :slight_smile:

[gaku-sei]

Thank you very much @bobzhang , i will experiment again with the ppx next week and let you know if that fixes it 😄

[dylanirlbeck]

@bobzhang Tailwind PPX author here. Do you know if there's anything PPX authors like myself could do to prevent this issue from happening on CircleCI? I'm not familiar with the concept of a spinlock, nor how it'd be implemented in Reason/OCaml, so some guidance here would be much appreciated. Thanks so much!