search

benmaier / netwulf

Interactive visualization of networks based on Ulf Aslak's d3 web app.
MIT License
294 stars 26 forks source link

KeyboardInterrupt starts expensive background process #4

Closed ulfaslak closed 5 years ago

ulfaslak commented 5 years ago

This is a weird one. When I KeyboardInterrupt in a Jupyter Notebook, a very CPU hungry Python process starts.

screen shot 2018-11-12 at 14 37 52

When I restart the kernel, it terminates. If I terminate it in the activity monitor, the Jupyter kernel becomes busy (no output in terminal).

Even weirder, if I launch a new visualization, IT STOPS.

benmaier commented 5 years ago

possibly fixed (forgot thread.join()). Can you try?

benmaier commented 5 years ago

hey @ulfaslak did you have time to test it?

ulfaslak commented 5 years ago

Sorry, baby fog, will try later!

benmaier commented 5 years ago

sure, no rush! baby's more important :)

On Nov 19, 2018, at 13:40, Ulf Aslak notifications@github.com wrote:

Sorry, baby fog, will try later!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benmaier/netwulf/issues/4#issuecomment-439879704, or mute the thread https://github.com/notifications/unsubscribe-auth/AKOzvP5_zi5acIHqo8XSGS1bjO_FhrVvks5uwqargaJpZM4YZq9D.

ulfaslak commented 5 years ago

Alright, it was fast enough to try. So I upgraded with pip to v0.0.6 (right one?), but the bug is still there.

benmaier commented 5 years ago

Haven't updated in pip yet, you'd have to install from a cloned repo ..

On Nov 19, 2018, at 13:43, Ulf Aslak notifications@github.com wrote:

Alright, it was fast enough to try. So I upgraded with pip to v0.0.6 (right one?), but the bug is still there.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benmaier/netwulf/issues/4#issuecomment-439880419, or mute the thread https://github.com/notifications/unsubscribe-auth/AKOzvHoDsYKiQpXnOPjrzouCc05a_zq_ks5uwqdWgaJpZM4YZq9D.

ulfaslak commented 5 years ago

ah on sec

ulfaslak commented 5 years ago

ok i'm not being very thorough checking if I installed it right here, but downloading the repo and running python setup.py install it installs v 0.0.7, which also imports successfully, but throws the error DistutilsFileError: cannot copy tree '/anaconda3/lib/python3.7/site-packages/netwulf-0.0.7-py3.7.egg/netwulf/js': not a directory at visualize(G)

benmaier commented 5 years ago

cloning probably did not include the submodule then. Can you try

git clone --recurse-submodules git@git....

On Nov 19, 2018, at 13:50, Ulf Aslak notifications@github.com wrote:

ok i'm not being very thorough checking if I installed it right here, but downloading the repo and running python setup.py install it installs v 0.0.7, which also imports successfully, but throws the error DistutilsFileError: cannot copy tree '/anaconda3/lib/python3.7/site-packages/netwulf-0.0.7-py3.7.egg/netwulf/js': not a directory at visualize(G)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benmaier/netwulf/issues/4#issuecomment-439882423, or mute the thread https://github.com/notifications/unsubscribe-auth/AKOzvPH9htkQ5g9OuITlX4fihVyELGjvks5uwqkbgaJpZM4YZq9D.

ulfaslak commented 5 years ago

Different behavior now. I interrupt and it doesn't stop, I interrupt it again and it throws error. nov-19-2018 13-57-46

benmaier commented 5 years ago

ok, off to the to-do pile then. thanks for checking!

On Nov 19, 2018, at 14:00, Ulf Aslak notifications@github.com wrote:

Different behavior now. I interrupt and it doesn't stop, I interrupt it again and it throws error. https://user-images.githubusercontent.com/6390218/48708582-461a8400-ec03-11e8-8a23-9740e34c0f4b.gif — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benmaier/netwulf/issues/4#issuecomment-439884934, or mute the thread https://github.com/notifications/unsubscribe-auth/AKOzvEnFp5qusrS7Hb_PU7X5wtmB_b0qks5uwqt7gaJpZM4YZq9D.

benmaier commented 5 years ago

is this fixed by now? 🤔

benmaier commented 5 years ago

can't reproduce

ulfaslak commented 5 years ago

This still happens to me. It's driving me crazy. Also, connected to this, the POST back to Python feature doesn't work. I'll look into this further and we can probably close this for now but jeeeeeesus....

benmaier commented 5 years ago

super weird ...

On Apr 9, 2019, at 06:07, Ulf Aslak notifications@github.com wrote:

Closed #4 https://github.com/benmaier/netwulf/issues/4.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/benmaier/netwulf/issues/4#event-2262607293, or mute the thread https://github.com/notifications/unsubscribe-auth/AKOzvIlNBxk4S91sPKphMwKyouONfuXLks5vfIJrgaJpZM4YZq9D.

ulfaslak commented 5 years ago

It seems to everything right down to line 274 if interactive.py where it hangs on thread.join().

benmaier commented 5 years ago

I reopen this ... you won't be the only one having this problem

benmaier commented 5 years ago

can you post the specific error message?

ulfaslak commented 5 years ago

This is the output when hitting "Post to Python" and then upon returning to the notebook (where the kernel is still busy), interrupting the kernel:

changing directory to /Users/ulfaslak/.netwulf
starting server here ... /Users/ulfaslak/.netwulf
127.0.0.1 09/Apr/2019 14:17:27 GET /?data=tmp_16a02081db3.json&config=config_tmp_16a02081db3.json HTTP/1.1 200 -
127.0.0.1 09/Apr/2019 14:17:28 GET /config_tmp_16a02081db3.json HTTP/1.1 200 -
127.0.0.1 09/Apr/2019 14:17:28 GET /tmp_16a02081db3.json HTTP/1.1 200 -
127.0.0.1 09/Apr/2019 14:17:29 POST /?data=tmp_16a02081db3.json&config=config_tmp_16a02081db3.json HTTP/1.1 200 -
Successfully posted network data to Python!
127.0.0.1 09/Apr/2019 14:17:31 POST /?data=tmp_16a02081db3.json&config=config_tmp_16a02081db3.json HTTP/1.1 200 -
stopping server ...
was asked to stop the server
deleted all files
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-6-a227c1684ca5> in <module>
      6     v['group'] = v['block']; del v['block']
      7 
----> 8 visualize(G, verbose=True)

/anaconda3/lib/python3.7/site-packages/netwulf/interactive.py in visualize(network, port, verbose, config)
    272         print('stopping server ...')
    273     server.stop_this()
--> 274     thread.join()
    275 
    276     posted_network_properties = server.posted_network_properties

/anaconda3/lib/python3.7/threading.py in join(self, timeout)
   1030 
   1031         if timeout is None:
-> 1032             self._wait_for_tstate_lock()
   1033         else:
   1034             # the behavior of a negative timeout isn't documented, but

/anaconda3/lib/python3.7/threading.py in _wait_for_tstate_lock(self, block, timeout)
   1046         if lock is None:  # already determined that the C code is done
   1047             assert self._is_stopped
-> 1048         elif lock.acquire(block, timeout):
   1049             lock.release()
   1050             self._stop()

KeyboardInterrupt:
ulfaslak commented 5 years ago

I'm on Python 3.7.3, and netwulf version 0.0.14

benmaier commented 5 years ago

can you try with the latest commit? I'm pretty sure it won't work but a guy can dream

ulfaslak commented 5 years ago

I like your spirit but no cigar. Also, this never runs if you don't interrupt (click Post to Python in webapp)

ulfaslak commented 5 years ago

Can it have something to do with the http.serve_forever() line? And that it excepts OSErrors? I tried removing the try-except clause, and nothing happened, but still as I understand it, thread.join() is ineffective because thread is still busy.

Recall that everything runs smoothly down to and including line 273: server.stop_this()

benmaier commented 5 years ago

Yeah, I was wondering the same. It's weird though, because self.server_close() in line 83 should definitely stop the process.

I'll look into it again in a few days, have to prepare a talk today :/

Here are some links I found which might contain a solution. Using this comment as a bookmark:

https://stackoverflow.com/questions/3788208/threading-ignores-keyboardinterrupt-exception

https://www.oreilly.com/library/view/python-cookbook/0596001673/ch06s03.html

https://github.com/agronholm/pythonfutures/pull/50

https://stackoverflow.com/questions/50891103/after-keyboardinterrupt-except-and-finally-blocks-are-not-executing

https://github.com/eventlet/eventlet/issues/230

http://paste.openstack.org/show/204506/

On Apr 9, 2019, at 06:36, Ulf Aslak notifications@github.com wrote:

Can it have something to do with the http.serve_forever() line? And that it excepts OSErrors? I tried removing the try-except clause, and nothing happened, but still as I understand it, thread.join() is ineffective because thread is still busy.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/benmaier/netwulf/issues/4#issuecomment-481232696, or mute the thread https://github.com/notifications/unsubscribe-auth/AKOzvCYk07v1wtlwvk0soSUW6TpgjjqDks5vfIkwgaJpZM4YZq9D.

ulfaslak commented 5 years ago

Cool thanks. No stress, just got hung up on this now...

benmaier commented 5 years ago

I've added a timeout to thread.join() -- can you try again with the new commit?

benmaier commented 5 years ago

I now added a functionality which stops serve_forever

ulfaslak commented 5 years ago

Sweet this works! Causes an error when doing a keyboardinterrupt in Jupyter, made a pull request that fixes this #13 .