run XMLRPC server in a separate thread by default

[GoogleCodeExporter]

run server in a separate thread so that administrator keeps an active
Python prompt, and can continue to manage the server through that Python
interface, e.g. run Python via screen so you can reconnect at any time to
the server process and manage it via new Python commands.

Original issue reported on code.google.com by cjlee...@gmail.com on 11 Sep 2008 at 4:26

[GoogleCodeExporter]

Attached you will find a patch which addresses this issue. What it does is 
pretty 
self-explanatory so let me just go over some more general issues:

 - since Python under Windows supports threading but not forking, it should now in 
principle be possible to run the XML-RPC server under Windows, should one be so 
inclined;

 - if the server is launched in the foreground from a non-interactive Python session 
it will of course shut down the moment the session terminates, i.e. quite 
likely 
almost immediately. I have added a warning about this, it doesn't however take 
into 
account whether the -i option has been given to the interpreter (in which case 
the 
warning doesn't apply, as the server happily keeps on running until the session 
is 
terminated by hand) or not. Is there some way of telling whether this flag has 
been 
set other than parsing argv?

 - I left the daemon mode as default rather than selectable as suggested in the 
description of the issue, as in my opinion this is the mode which should be 
used in 
production running (especially once we've added management commands to the 
server, 
which will make it unnecessary to manipulate it by hand from the Python command 
line), plus changing this could break automated server-launcher scripts during 
an 
upgrade from earlier versions of Pygr (see the second remark).

Comments are welcome!

Original comment by mare...@gmail.com on 17 Feb 2009 at 7:22

Attachments:

• pygrData-xmlrpcServer-threadedFG.patch

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I don't really understand the original question, when it says: "run Python via 
screen
so you can reconnect at any time to the server process and manage it via new 
Python
commands."

What is an example of a new Python command that one would want to use to manage 
the
server process after it has started?

Original comment by istvan.a...@gmail.com on 17 Feb 2009 at 8:18

[GoogleCodeExporter]

@Istvan: I suggested using the screen program to reconnect to the Python 
interpreter
session running the XMLRPC server, as a general way of being able to manage the
server (make changes, add new data sources, etc.) while the server is running.  

THE PROBLEM: Currently, the server always detaches itself from the console as a
background process, so even if you started it from an interactive Python 
interpreter,
you lose that connection and have no way of managing the server.  Since we also 
had
no secure XMLRPC command mechanism either, this left us with no way at all to 
manage
the server.  If you needed to make any change, no matter how trivial, you'd 
have to
kill the server and restart it with whatever changes you wanted.

While I like the idea of a general and secure XMLRPC command interface for 
managing
the server, I think this will take some time to design and develop.  We haven't 
even
had a discussion about what commands people would want.  At any rate, this 
seems to
fall into the category of pygr.Data reform, which is scheduled for the 1.0 
release...
some distance in the future.

By contrast, it would be trivial to make the following change for the 0.8 
release,
giving a secure, general way of managing the XMLRPC server:

- start the server in a separate thread, so that the original Python interpreter
remains responsive even while the server is running.  If the process is an
interactive interpreter, don't detach from the console.  This would allow the 
user to
continue to issue Python commands that could control / update the server in any
desired way.  Note that this isn't true multi-threading in the sense that Titus 
was
warning could cause trouble; it merely means retaining an interactive Python
interpreter command line once the XMLRPC server is started.

- Recommend that users initially launch this interactive session within the UNIX
program screen, so that they can detach and re-attach their terminal to this
interactive session whenever they like.  Screen makes this really convenient.

- only the original user who launched the interactive session can re-connect to 
it
using screen, so it remains secure.

Original comment by cjlee...@gmail.com on 17 Feb 2009 at 9:28

[GoogleCodeExporter]

Re-posted the patch in git format-patch format, for merging convenience.

Original comment by mare...@gmail.com on 21 Feb 2009 at 1:23

• Changed state: FixedNeedsReview

Attachments:

• 0001-Made-it-possible-to-run-the-pygr.Data-XML-RPC-server.patch

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Original comment by mare...@gmail.com on 4 Mar 2009 at 7:34

[GoogleCodeExporter]

Original comment by cjlee...@gmail.com on 4 Mar 2009 at 11:51

[GoogleCodeExporter]

Original comment by mare...@gmail.com on 13 Mar 2009 at 1:00

• Added labels: reviewby-deepreds

[GoogleCodeExporter]

I couldn't find a way how to check whether user give "-i" or not. I removed 
that part and changed a few words in 
warning message. Functionality has bee tested and reviewed. Now, XMLRPC server 
in biodb2:5000 is running as 
daemonized server (default, demonize = True).

Original comment by deepr...@gmail.com on 13 Apr 2009 at 2:01

Attachments:

• 0001-XMLRPC-server-do-not-check-sys-ps1.patch

[GoogleCodeExporter]

Hmm, I'm not sure whether removing the ps1 check is a good idea... Let me 
explain. 
AFAIK there are three scenarios of running the server with demonise=False:
 (1) start an interactive Python session, type in the invocation; this will keep the 
server running as long as the session isn't manually terminated;
 (2) 'python scriptname' or something like that, in which case the (non-interactive) 
session is terminated as soon as EOF is reached in the script input - taking 
the 
server with it right after it started;
 (3) 'python -i scriptname' or something like that, in which case the non-
interactive session turns interactive as soon as EOF is reached in the script 
input 
- keeping the server running as long as the interactive session isn't manually 
terminated.

According to Python mailing list, _hasattr(sys, 'ps1')_ is a de-facto standard 
way 
of detecting whether a Python session is interactive or not, thus making sure 
the 
warning message is not printed in case (1). It doesn't work for (3) because ps1 
is 
only set AFTER the script has been run. Thus, we end up with the following 
behaviour:
 (1) run without hassle - as expected;
 (2) warn the user about dying, then die - as expected;
 (3) warn the user about dying, then NOT die - a false negative

Removing the check on the other hand means we've got a warning produced every 
time 
the server starts in non-daemon mode, which means we've got two false negatives 
rather than one.

Original comment by mare...@gmail.com on 13 Apr 2009 at 7:23

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I've figured out how to check for -i! Attached you will find a patch which 
implements this. The catch? The relevant objects have only been added to Python 
in 
version 2.6 so running under older version will keep giving a false negative 
for (3).

Original comment by mare...@gmail.com on 13 Apr 2009 at 8:03

Attachments:

• 0001-If-running-under-Python-2.6-and-up-check-if-i-has.patch

[GoogleCodeExporter]

OK. Now I understand better what is happening. What about adding more comment 
or 
warning message for these cases?

Original comment by deepr...@gmail.com on 14 Apr 2009 at 12:48

• Changed state: ReviewedNeedsCommit

[GoogleCodeExporter]

These wouldn't hurt... Then again, I've asked the Python Usenet group about a 
2.3-
compatible way of querying -i - maybe we will be able to resolve the problem 
properly.

PS. Since new patches have been provided I have spawned a new GitHub branch 
dedicated to the issue: http://github.com/ctb/pygr/commits/issue037

Original comment by mare...@gmail.com on 14 Apr 2009 at 1:13

[GoogleCodeExporter]

Chris has pushed the code into the master and the dedicated branch has been 
deleted. 
Closing the issue.

Original comment by mare...@gmail.com on 14 Apr 2009 at 9:20

• Changed state: Closed