search

knennigtri / merge-markdown

A tool to take in a list of markdown files and merge them with optional HTML/PDF output
76 stars 9 forks source link

Race condition ? - no such file or directory #44

Closed Techbrunch closed 2 years ago

Techbrunch commented 2 years ago

Hello,

It appears that there is some kind of race condition when trying to merge files.

This does not happen every time but often enough.

You can reproduce the issue with this manifest (all the markdown files are empty):

---
input:
  1.md: {noYAML: true}
  2.md: {noYAML: true}
  3.md: {noYAML: true}
  4.md: {noYAML: true}
  5.md: {noYAML: true}
  6.md: {noYAML: true}
  7.md: {noYAML: true}
  8.md: {noYAML: true}
  9.md: {noYAML: true}
  10.md: {noYAML: true}

TOC: false
output: bug.md
---

output:

❯ merge-markdown
No -m argument given. Using default: manifest.[md|yaml|yml|json]
Found manifest to use: manifest.yml
*********.//1.md*********
1.md.temp added to merge list
*********.//2.md*********
2.md.temp added to merge list
*********.//3.md*********
3.md.temp added to merge list
*********.//4.md*********
4.md.temp added to merge list
*********.//5.md*********
5.md.temp added to merge list
*********.//6.md*********
6.md.temp added to merge list
*********.//7.md*********
7.md.temp added to merge list
*********.//8.md*********
8.md.temp added to merge list
*********.//9.md*********
9.md.temp added to merge list
*********.//10.md*********
10.md.temp added to merge list
++++++++++++++++++++
List of files to merge:
    .//1.md.temp
    .//2.md.temp
    .//3.md.temp
    .//4.md.temp
    .//5.md.temp
    .//6.md.temp
    .//7.md.temp
    .//8.md.temp
    .//9.md.temp
    .//10.md.temp
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: ENOENT: no such file or directory, open '/private/tmp/poc/5.md.temp'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/private/tmp/poc/5.md.temp'
}

Node.js v18.8.0

Thanks for the tool !

knennigtri commented 2 years ago

@Techbrunch what version are you using? I just tested this with the latest 0.9.0 and could not reproduce.

Techbrunch commented 2 years ago

I'm running version 0.9.0, if it works on your side then I'm affraid this might be due to my config: MacBook Pro M1

knennigtri commented 2 years ago

Hmm That's the same system I'm running with Monterey. Granted, I just realized I'm using Node.js 14.17.5 so I'll update and see if that produces it. In the meantime, can you run:

DEBUG=* merge-markdown

and then give me the debug messages right before the error as well?

knennigtri commented 2 years ago

I just upgraded to node v18.9.0 and I still am not able to produce the error

merge-markdown
No -m argument given. Using default: manifest.[md|yaml|yml|json]
Found manifest to use: manifest.md
*********.//1.md*********
1.md.temp added to merge list
*********.//2.md*********
2.md.temp added to merge list
*********.//3.md*********
3.md.temp added to merge list
*********.//4.md*********
4.md.temp added to merge list
*********.//5.md*********
5.md.temp added to merge list
*********.//6.md*********
6.md.temp added to merge list
*********.//7.md*********
7.md.temp added to merge list
*********.//8.md*********
8.md.temp added to merge list
*********.//9.md*********
9.md.temp added to merge list
*********.//10.md*********
10.md.temp added to merge list
++++++++++++++++++++
List of files to merge:
    .//1.md.temp
    .//2.md.temp
    .//3.md.temp
    .//4.md.temp
    .//5.md.temp
    .//6.md.temp
    .//7.md.temp
    .//8.md.temp
    .//9.md.temp
    .//10.md.temp
Techbrunch commented 2 years ago

Using Debug=*

❯ DEBUG=* merge-markdown
No -m argument given. Using default: manifest.[md|yaml|yml|json]
  index Found default manifest.[md|yaml|yml|json] +0ms
Found manifest to use: manifest.yml
  index:manifest {
  index:manifest   "input": {
  index:manifest     "1.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "2.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "3.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "4.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "5.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "6.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "7.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "8.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "9.md": {
  index:manifest       "noYAML": true
  index:manifest     },
  index:manifest     "10.md": {
  index:manifest       "noYAML": true
  index:manifest     }
  index:manifest   },
  index:manifest   "TOC": false,
  index:manifest   "output": "bug.md"
  index:manifest } +0ms
*********.//1.md*********
  merge Rewriting Relative Asset Paths +0ms
  merge:relLinks 0 relative paths updated +0ms
  merge --Apply global manifest OPTIONS-- +1ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +0ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +0ms
  merge:linkcheck FILE: .//1.md +3ms
1.md.temp added to merge list
*********.//2.md*********
  merge Rewriting Relative Asset Paths +4ms
  merge:relLinks 0 relative paths updated +4ms
  merge --Apply global manifest OPTIONS-- +0ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +4ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +1ms
  merge:linkcheck FILE: .//2.md +2ms
2.md.temp added to merge list
*********.//3.md*********
  merge Rewriting Relative Asset Paths +2ms
  merge:relLinks 0 relative paths updated +2ms
  merge --Apply global manifest OPTIONS-- +0ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +2ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +0ms
  merge:linkcheck FILE: .//3.md +1ms
3.md.temp added to merge list
*********.//4.md*********
  merge Rewriting Relative Asset Paths +1ms
  merge:relLinks 0 relative paths updated +1ms
  merge --Apply global manifest OPTIONS-- +0ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +1ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +0ms
  merge:linkcheck FILE: .//4.md +1ms
4.md.temp added to merge list
*********.//5.md*********
  merge Rewriting Relative Asset Paths +1ms
  merge:relLinks 0 relative paths updated +1ms
  merge --Apply global manifest OPTIONS-- +0ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +1ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +0ms
  merge:linkcheck FILE: .//5.md +1ms
5.md.temp added to merge list
*********.//6.md*********
  merge Rewriting Relative Asset Paths +1ms
  merge:relLinks 0 relative paths updated +1ms
  merge --Apply global manifest OPTIONS-- +0ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +1ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +0ms
  merge:linkcheck FILE: .//6.md +1ms
6.md.temp added to merge list
*********.//7.md*********
  merge Rewriting Relative Asset Paths +1ms
  merge:relLinks 0 relative paths updated +1ms
  merge --Apply global manifest OPTIONS-- +0ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +1ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +0ms
  merge:linkcheck FILE: .//7.md +1ms
7.md.temp added to merge list
*********.//8.md*********
  merge Rewriting Relative Asset Paths +1ms
  merge:relLinks 0 relative paths updated +1ms
  merge --Apply global manifest OPTIONS-- +1ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +2ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +1ms
  merge:linkcheck FILE: .//8.md +0ms
8.md.temp added to merge list
*********.//9.md*********
  merge Rewriting Relative Asset Paths +0ms
  merge:relLinks 0 relative paths updated +1ms
  merge --Apply global manifest OPTIONS-- +0ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +0ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +0ms
  merge:linkcheck FILE: .//9.md +1ms
9.md.temp added to merge list
*********.//10.md*********
  merge Rewriting Relative Asset Paths +1ms
  merge:relLinks 0 relative paths updated +1ms
  merge --Apply global manifest OPTIONS-- +0ms
  merge --Apply file specific OPTIONS-- +0ms
  merge [OPTION] Remove YAML... +0ms
  merge:yaml S(-1)E(-1)✓'d(0)T(1) +1ms
  merge:yaml No YAML found for removal +0ms
  merge --Create/Update linkcheck file-- +0ms
  merge:linkcheck Folder to be linkchecked: file:/private/tmp/poc/ +0ms
  merge:linkcheck FILE: .//10.md +1ms
10.md.temp added to merge list
++++++++++++++++++++
List of files to merge:
    .//1.md.temp
    .//2.md.temp
    .//3.md.temp
    .//4.md.temp
    .//5.md.temp
    .//6.md.temp
    .//7.md.temp
    .//8.md.temp
    .//9.md.temp
    .//10.md.temp
  merge Creating single file +1ms
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: ENOENT: no such file or directory, open '/private/tmp/poc/10.md.temp'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/private/tmp/poc/10.md.temp'
}

Node.js v18.8.0
Techbrunch commented 2 years ago

I was able to reproduce it using Docker:

Dockerfile:

FROM node

RUN npm install -g @knennigtri/merge-markdown

ENTRYPOINT ["merge-markdown"]

The manifest:

---
input:
  1.md: {noYAML: true}
  2.md: {noYAML: true}
  3.md: {noYAML: true}
  4.md: {noYAML: true}
  5.md: {noYAML: true}

TOC: false
output: bug.md
---

Execution

❯ docker run -v $(pwd):/tmp merge-markdown -m /tmp
Found manifest to use: /tmp/manifest.yml
*********/tmp/1.md*********
1.md.temp added to merge list
*********/tmp/2.md*********
2.md.temp added to merge list
*********/tmp/3.md*********
3.md.temp added to merge list
*********/tmp/4.md*********
4.md.temp added to merge list
*********/tmp/5.md*********
5.md.temp added to merge list
++++++++++++++++++++
List of files to merge:
    /tmp/1.md.temp
    /tmp/2.md.temp
    /tmp/3.md.temp
    /tmp/4.md.temp
    /tmp/5.md.temp
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: ENOENT: no such file or directory, open '/tmp/1.md.temp'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/tmp/1.md.temp'
}

Node.js v18.9.0
knennigtri commented 2 years ago

Try installing this bugfix and see if that solves it. 

npm install --save @knennigtri/merge-markdown@bugfix

Worst case, there is now an option that allows you to remove linkcheck (where the error is):

merge-markdown --nolinkcheck
Techbrunch commented 2 years ago

Yes it works, thanks !

knennigtri commented 2 years ago

Cool. Just keep using the bugfix for now. It will come out with 1.0.0 in the next few weeks.