lookup optimizations and reporting

[GoogleCodeExporter]

1. Think about lookup.py reporting (right now, stderr messages from low
level scanners are mixed up). Reports should not be very long (should not
include every host), but must be informative enough.

2. Optimization: use `python -m cProfile lookup.py runall` to discover
long-running code. Preliminary analysis shows that port scanning is not
enough fast under Windows.

Original issue reported on code.google.com by radist...@gmail.com on 10 Feb 2010 at 2:26

[GoogleCodeExporter]

>port scanning is not enough fast under Windows.

I would be very surprised if that set(sockets) - set(finished_sockets) after 
each
select was fast.
I was thinking about switching from select() to poll()/epoll() but are they 
supported
in Windows? I'm not against os-specific code here, but I have no idea at all 
what
should be used instead of select() in Windows.

Original comment by ruslan.savchenko on 10 Feb 2010 at 2:57

[GoogleCodeExporter]

Does lookuper really need to scan new shares availability and connectivity each 
run?
Or it could just check existing shares each run and check new shares sometimes?

Original comment by radist...@gmail.com on 10 Feb 2010 at 9:36

[GoogleCodeExporter]

I believe the main job of the lookuper is to find new shares. Garbage 
collection is
also nice, but I don't care about it much.

Original comment by ruslan.savchenko on 10 Feb 2010 at 9:55

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

>I was thinking about switching from select() to poll()/epoll() but are they 
supported
>in Windows? I'm not against os-specific code here, but I have no idea at all 
what
>should be used instead of select() in Windows.

In our case 85% of running time (on Windows) is a stall causing by waiting
unfinishable sockets. Maybe there are some flags to tell windows to close that
sockets faster?

Original comment by ruslan.savchenko on 11 Feb 2010 at 2:54

[GoogleCodeExporter]

scantype discovering should be processed right after discovering open ports
(so scan_all_hosts code should be dublicated somewhere in Lookup::commit)

Original comment by radist...@gmail.com on 11 Feb 2010 at 6:06

[GoogleCodeExporter]

> scantype discovering should be processed right after discovering open ports
done

> 1. Think about lookup.py reporting ...
At most required cases, logging is enough

> 2. Optimization ...
Seems like lookuping times is not our first priority optimization task. Also, 
port
scanning works fine enough with not too small max_connections values (pinging 
times
for 282 shares with ~150 online shares are less then minute).
More productivity could be obtained by simultaneous looking up in different 
networks.

So, if no objection will appear today, I'll close this issue.

Original comment by radist...@gmail.com on 15 Feb 2010 at 8:26

[GoogleCodeExporter]

Original comment by radist...@gmail.com on 16 Mar 2010 at 6:17

• Changed state: Fixed