Race condition with parallel builds

[yfyf]

Hi,

the writes / deletes of the piqi.piqi.pb file during calls to piqic-erlang cause parallel compilations (happening within the same directory) to fail.

A super hacky fix would be to just parametrise the self-spec filename by the filename of the thing being compiled. Alternatively, I see there's a TODO about sending it to stdin of piqi compile, is that even possible?

However, I think there's a third and much more sane solution: compile the self-spec once, when compiling piqi-erlang itself. IMHO if you are bumping your system's piqi binary, you will in any case take care to re-compile things like piqi-erlang. This would also be a decent performance improvement.

What do you think?

[alavrik]

Yep, this is a TODO. I wasn't expecting it to cause problems though! So thank you for reporting this.

Sending piqi_piqi:piqi() to piqi compile is exactly the intended way of doing this. You just need to pass --self-spec - instead of --self-spec piqi.piqi.pb. It it a trivial change by itself. The only reason I haven't done this earlier is because sending data to stdin of another spawned program in Erlang requires writing 15 lines of code that starts with Port = open_port({spawn_executable, ... :) Which is essentially what eunit_lib:command() does under the hood but it doesn't expose the interface for sending data to the spawned command.

A patch would be appreciated or I can take care of it myself a bit later.

[yfyf]

Thanks, I might patch this myself some time later, but wanted to ask you about the preferred way and let you know about the issue.

[yfyf]

I tried to do this but then remembered why I was reluctant to implement it at the beginning: interacting with OS processes via Erlang ports is a major PITA, because you can't close the stdin stream without closing the port (and thus losing its output) and you can't separate stderr from stdout.

There are several possible solutions, none of which I like:

• (super nasty) introduce a wrapper shell script which executes the piqi binary and tags it's stderr / stdout with string prefixes and is able to receive an explicit "EOF" message, read the output from piqi compile and pass it back.
• (overkill) use a proper OS process management library, such as https://github.com/saleyn/erlexec
• (easy hack) just avoid the race condition by using non-clashing temp file names (either via something like erlang:phash2(make_ref()) or mktemp)

What would you prefer?

[alavrik]

Ah. Forgot about this Erlang's highly annoying limitation. Temp files as a workaround should be totally fine.