search

Interlisp / medley

The main repo for the Medley Interlisp project. Wiki, Issues are here. Other repositories include maiko (the VM implementation) and Interlisp.github.io (web site sources)
https://Interlisp.org
MIT License
377 stars 19 forks source link

Getting reliable results from CREATE-PROCESS-STREAM #819

Closed rmkaplan closed 1 year ago

rmkaplan commented 2 years ago

I am using CREATE-PROCESS-STREAM to execute the various git commands. Until a week or so ago, this seemed to be working reliably.

I was executing a git command, then reading the results that that command printed out (log information, file contents, etc.) from the process stream, one byte at a time until the end-of-file was reached (NIL BIN after setting ENDOFSTREAMOP to NILL.

This stopped working: it began to hang forever on the BIN at the end (I think).

So I thought of a different appproach: Tell the git command to redirect its output to a tmp/ file whose name is known also to Medley. Then simply read the bytes from the temp file, an ordinary file.

This also seems to be flaky. It appears that there is a timing issue, that CREATE-PROCESS-STREAM returns before the action is complete. Putting in a DISMISS for a couple of hundred milliseconds seems to help, but not in all cases. And of course, when I run the git command (with the redirect) in terminal shell, it always is complete and correct.

My Unix knowledge about this is pretty weak. What is the best way of getting results reliably from the execution of a shell command from Medley?

rmkaplan commented 2 years ago

I think there is now definitely a race condition that I didn't observe before. If I don't put a delay of at least a second before trying to read from the process stream, the result is unreliable. For example even getting the list of pr's for the prc menu returns NIL if I don't put in that delay. And many of the file fetches come up empty.

I really should only have to wait until the process is complete. Is the a way of telling that? Or is what I'm observing really due to some other change that happened.

masinter commented 2 years ago

i think there's might be a problem using ";" between shell commands -- it forks separate processes and probably the shell stuff in maiko might not expect that. Try using && instead gh fetch && git show && echo STOP Writing local files in /tmp/interlisp/medley/ae123df3/ would be a lot faster

rmkaplan commented 2 years ago

I was only using ; between the cd command and the actually command.

Replacing ; with && doesn’t change the behavior, e.g. the pr menu is empty until the delay gets up to about 900 msec.

Redirecting output to Unix /tmp/xx doesn’t seem to make a difference.

On Jul 6, 2022, at 11:49 PM, Larry Masinter @.***> wrote:

i think there's might be a problem using ";" between shell commands -- it forks separate processes and probably the shell stuff in maiko might not expect that. Try using && instead gh fetch && git show && echo STOP Writing local files in /tmp/interlisp/medley/ae123df3/ would be a lot faster

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/819#issuecomment-1177160906, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJIXXSV736FRBN2J5F3VSZ4ZDANCNFSM52W7XW4A. You are receiving this because you authored the thread.

rmkaplan commented 2 years ago

I think I now understand what you are recommending. If I redirect to /tmp/, then if the process stream doesn’t return with just STOP (or COMPLETED), then the command failed. Otherwise, the tmp file contains either the results of the command, or the requested file. Seems to work.

Can I assume that the git command will signal failure in the case when the process stream would have had the message “fatal: …” ? Or do I still have to check for that? I.e., cause an error if the first line of the redirect is “fatal: “ even when the process stream says COMPLETED.

There is a little issue about the external format. If the results are to be interpreted as the git output of a command (like log) and not the contents of a file, then the redirect is presumably in the current LANG format. That was the format of the process stream, given recent changes, but that information gets lost when it is passed up to the output interpretation level. I think we need a separate little function, maybe in EXTERNALFORMAT, that simply returns the current LANG format. UNIXCOMM would call it.

(This may or may not be relevant in the case of retrieving a file: file retrieval is binary, our Lisp files know their own format, and we have other heuristics about files of other types (currently XCCS).)

On Jul 7, 2022, at 12:05 AM, Ron Kaplan @.***> wrote:

I was only using ; between the cd command and the actually command.

Replacing ; with && doesn’t change the behavior, e.g. the pr menu is empty until the delay gets up to about 900 msec.

Redirecting output to Unix /tmp/xx doesn’t seem to make a difference.

On Jul 6, 2022, at 11:49 PM, Larry Masinter @. @.>> wrote:

i think there's might be a problem using ";" between shell commands -- it forks separate processes and probably the shell stuff in maiko might not expect that. Try using && instead gh fetch && git show && echo STOP Writing local files in /tmp/interlisp/medley/ae123df3/ would be a lot faster

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/819#issuecomment-1177160906, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJIXXSV736FRBN2J5F3VSZ4ZDANCNFSM52W7XW4A. You are receiving this because you authored the thread.

rmkaplan commented 2 years ago

Turns out that some of the git output goes to the output stream and some goes to the error stream. If I redirect both to tmp/, then everything is interpretable. The process stream only signals whether the command completed successfully.

One level up, my solution is to say that if both streams are non-empty, then the output stream is passed as the result and the error stream is copied to T. Otherwise, I pass up as the result whichever stream is non-empty.

masinter commented 2 years ago

I'm getting an error when I load GITFNS in my init that NIL isn't an external format (under \EXTERNALFORMAT) I think. And loadup-db.sh fails. I'm trying to track down when the change happened

It surely seems wrong to set the projects when you load GITFNS anyway.

ShellCommand("pwd") fails

masinter commented 2 years ago

When I roll back to f86be45834 don't get the error... not sure when it was introduced.

rmkaplan commented 2 years ago

I’m reworking a lot, waiting for other PR approvals before I put out the newer GITFNS.

On Jul 8, 2022, at 10:45 AM, Larry Masinter @.***> wrote:

When I roll back to f86be45 https://github.com/Interlisp/medley/commit/f86be45834fe441ac31c786656960cd714e22e2a don't get the error... not sure when it was introduced.

— Reply to this email directly, view it on GitHub https://github.com/Interlisp/medley/issues/819#issuecomment-1179228623, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQSTUJKT6FGNXDWJWBQ4D6LVTBSLPANCNFSM52W7XW4A. You are receiving this because you authored the thread.

masinter commented 1 year ago

this all seems to be working well... is there still a problem? Reopen if you think so.