Closed klmr closed 7 years ago
blahah
commented
7 years ago Oops, sorry about this!
ScienceFair doesn't use bittorrent but it does use UTP hole-punching and attempts to find peers via a mainline DHT so a small amount of the traffic might look like bittorent.
Can you give me any details of the institution? I'd like to follow up with them to see what's happening specifically.
cc @mafintosh
max-mapper
commented
7 years ago fwiw I don't think we've seen an example in the wild of anything blocking the DHT UDP messages... I think because its a very low amount of traffic. however it could definitely be done if a router did deep packet inspection and was looking specifically for DHT messages
blahah
commented
7 years ago @maxogden I'm now in contact with the sysadmin thanks to @klmr, and he's going to investigate tomorrow. I'll update here when I know more.
kdm9
commented
7 years ago FWIW, at my institution sciencefair hangs at 0% while syncing initial data. I can provide further debugging info if required (& will try at home tonight)
blahah
commented
7 years ago @kdmurray91 sorry about that - any additional info about the connection and network would be useful. If you are able to join the chat (irc.freenode.net #sciencefair) while at your institute I can guide you through debugging
klmr
commented
7 years ago @kdmurray91 This is the same behaviour that I am also experiencing, so the cause might be related (= blocked by firewall).
HBossier
commented
7 years ago Same behavior here!
/Applications/ScienceFair.app/Contents/MacOS/ScienceFair ; exit; GVA info: Successfully connected to the Intel plugin, offline Gen75 App is ready [14:29:36:0809] [info] Checking for update [14:29:38:0401] [info] Update for version 1.0.5 is not available (latest version: 1.0.5, downgrade is disallowed.
boazmohar
commented
7 years ago Same for me, at HHMI's Janelia Reaserch Campus. I am not sure if anything is blocked but the application is stuck on Syncing initial data.
blahah
commented
7 years ago @HBossier @boazmohar this could be a network issue or it could be an error that has interrupted the sync.
If those affected could take a look at the devtools console to see if there are any errors, that would help. To open devtools:
HBossier
commented
7 years ago I don't really see any error in the devtools console. There is one line in the Verbose output:
[Violation] Added non-passive event listener to a scroll-blocking 'mousewheel' event. Consider marking event handler as 'passive' to make the page more responsive.`
In the OSX console, I can read:
30/06/17 17:08:06,243 ScienceFair Helper[4258]: GVA info: preferred scaler idx 0
Which is not really helpful either I suppose.
blahah
commented
7 years ago @HBossier that is helpful (although I agree it doesn't seem like it is)! That tells me your issues are very likely network related and not errors in the app.
Can you describe the network setup a bit? i.e.:
And if you're comfortable in the command-line and have node installed, please try this and post the results:
npm install -g p2p-test
p2p-testI have left it on for the past 6 hours and I am at 3% right now. In the console I only see:
C:\Program Files\ScienceFair\resources\app.asar\node_modules\raven-js\src\console.js:26 Content server serving C:\Users\moharb\.sciencefair\datasources at 8000If I refresh I also get:
C:\Program Files\ScienceFair\resources\app.asar\node_modules\search-index-adder\lib\delete.js:70 Uncaught TypeError: Cannot convert undefined or null to object
at Function.keys (<anonymous>)
at DBEntries._transform (C:\Program Files\ScienceFair\resources\app.asar\node_modules\search-index-adder\lib\delete.js:70)
at DBEntries.Transform._read (_stream_transform.js:167)
at DBEntries.Transform._write (_stream_transform.js:155)
at doWrite (_stream_writable.js:329)
at writeOrBuffer (_stream_writable.js:315)
at DBEntries.Writable.write (_stream_writable.js:241)
at DocVector.ondata (_stream_readable.js:557)
at emitOne (events.js:96)
at DocVector.emit (events.js:191)
at readableAddChunk (_stream_readable.js:178)
at DocVector.Readable.push (_stream_readable.js:136)
at DocVector.Transform.push (_stream_transform.js:128)
at C:\Program Files\ScienceFair\resources\app.asar\node_modules\search-index-adder\lib\delete.js:47
at dispatchError (C:\Program Files\ScienceFair\resources\app.asar\node_modules\levelup\lib\util.js:25)
at C:\Program Files\ScienceFair\resources\app.asar\node_modules\levelup\lib\levelup.js:228I also did the p2ptest:
Network tested. Thank you for participating.
{ time: '2017-06-30T19:11:44.244Z',
description: '',
holePunchable: false,
pings:
[ { port: 15179, host: '206.241.0.254' },
{ port: 32513, host: '206.241.0.254' } ] }thanks @boazmohar. So there are two issues here. One is a bug in a dependency that will be worked around in the next release, the other is that your network is blocking UTP hole punching which might make connecting to peers difficult. A workaround for this will take a bit longer but should land in a release early next week.
step21
commented
7 years ago To add to this - apart from being blocked by firewalls, also on my very unencumbered home connection the packaged app failed to sync, but the dev app (from github) synced. Console says this:
blahah
commented
7 years ago Thanks for reporting this @step21 - as you've discovered, this bug has been fixed in the master branch but hasn't been released yet. A new release will be out in a few hours and should auto-update when you launch the app, then sync should work (at least, if this was the only bug ^_^)
p.s. hope you don't mind that I made the error trace in your comment collapsible (with <details>...</details>) to increase readability of the thread.
step21
commented
7 years ago Sure, no problem. Getting back to the thread, I think firewalls are quite a big problem. 1. because policies are often unknown and 2. because many institutions simply block everything that is somehow p2p. and 3. because it is hard/impossible to change this, even when necessary for research.
blahah
commented
7 years ago @step21 I agree, I have been thinking about this and have a rough plan:
In general dat does not work the same as most p2p tools; only some optional parts of the protocol might conceivably trigger such blocks. We should try to detect situations where those things are blocked/throttled if possible, and switch them off.
In cases where we can't connect well enough to reasonably use the p2p network at all, we will have stable nodes for every datasource which seed over HTTPS as well as the p2p protocols. Clients will automatically fall back to the HTTPS seeds.
Eventually, I'd like to make it so any client can volunteer to be a relay node, to help those behind firewalls join the swarm, acting something like a VPN but exclusively for other ScienceFair users and only for ScienceFair traffic.
blahah
commented
7 years ago To this end, I started work on sciencefair-doctor earlier today, which is essentially dat-doctor but wrapped in an electron app instead of node, and set up the same as ScienceFair. The idea is that this will become a module we can include in ScienceFair to detect and diagnose network issues, resolving them to specific configuration changes that need to be applied.
HBossier
commented
7 years ago Aight, So to give you some background information:
blahah
commented
7 years ago I tried to run the p2p-test. But after 5 minutes of waiting, there was no response.
@mafintosh o/ any idea what that means?
mafintosh
commented
7 years ago @blahah yup, that all non http ports are blocked which explains the sync issue.
mafintosh
commented
7 years ago non http or udp. perhaps we should test that tcp is also blocked for non http
step21
commented
7 years ago @HBossier You could try connecting through an alternative Wifi Network. At my institution at least, Eduroam has very few privileges. For example you cannot use it to print, but you must use another, institution specific wifi. Maybe it is the same for you and that network could be more privileged.
@blahah sounds like a plan.
HBossier
commented
7 years ago Unfortunately, eduroam is the only available network at the moment. I could try on cable, but I will have to contact our IT department to open ethernet ports for that. :)
step21
commented
7 years ago What how come you do not have open ethernet ports? At least in library and offices we always have those...
blahah
commented
7 years ago To all in this thread: tomorrow me and @mafintosh (who designed and made the underlying p2p protocols and code) are going to try to diagnose all these network issues so we can design the workarounds.
If anyone is willing and able to help, we are looking for volunteers who can:
Those who can do that, please just add a thumbs up emoji to this post (or comment if you'd like more details or to suggest anything) :)
HBossier
commented
7 years ago I have quite a busy schedule tomorrow, but will try to help as much as possible.
Immediately upon starting the application1, my computer’s internet access got blocked by the institute firewall.
I don’t know any technical specifics about the firewall. The one thing I know is that it’s used, amongst other things, to enforce the institute ”internet acceptable use”. This includes:
The affected desktop OS Is macOS 10.12.5 (though this shouldn’t matter).
1 Unfortunately I neglected to take a screenshot; roughly, the screen is showing a progress bar at 0%, with a message regarding “initial syncing”.