Sample memento doesn't work for me

[akavel]

A screencast:

and a copy of the initial and final ipwb stacktraces (they all seem to look the same to the naked eye):

C:\dnload\ipfs-etc>ipwb replay
This version of ipwb is outdated. Please run pip install --upgrade ipwb.
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
 * Running on http://localhost:5000/ (Press CTRL+C to quit)
Retrieving URI-Ms from c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
127.0.0.1 - - [08/Dec/2017 19:43:12] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:12] "GET /webui/webui.css HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:12] "GET /webui/webui.js HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET /webui/logo.png HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET /daemon/status HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET /config/openEndedPlaceHolder HTTP/1.1" 200 -
Retrieving URI-Ms from c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET /webui/favicons/apple-touch-icon.png HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET /webui/favicons/favicon-16x16.png HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET /webui/webui.css HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET /webui/webui.js HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 19:43:13] "GET /webui/daemonController.js HTTP/1.1" 200 -
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 64204)
Traceback (most recent call last):
  File "c:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "c:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "c:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "c:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "c:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "c:\python27\lib\site-packages\ipwb\replay.py", line 398, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "c:\python27\lib\site-packages\ipwb\replay.py", line 532, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 64206)
Traceback (most recent call last):
  File "c:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "c:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "c:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "c:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "c:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "c:\python27\lib\site-packages\ipwb\replay.py", line 398, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "c:\python27\lib\site-packages\ipwb\replay.py", line 532, in show_uri
    sys.exit()
SystemExit

[...]

----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 64222)
Traceback (most recent call last):
  File "c:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "c:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "c:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "c:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "c:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "c:\python27\lib\site-packages\ipwb\replay.py", line 398, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "c:\python27\lib\site-packages\ipwb\replay.py", line 532, in show_uri
    sys.exit()
SystemExit
----------------------------------------

[machawk1]

Oh, that's really not good. I must have broken something along the way that is not yet being tested in the Travis config.

I will look into this further, @akavel . Are you able to replicate this with the latest build? A git pull will suffice as a substitute for the instructions of using pip.

[akavel]

(Please note that ^ is an escape char on windows, as \ on Linux, so I'm escaping it as ^^ below.)

C:\dnload\ipfs-etc\ipwb>git log HEAD^!
fatal: ambiguous argument 'HEAD!': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

C:\dnload\ipfs-etc\ipwb>git log HEAD^^!
commit f17677f6a194365bd6bd7e0a20c702f1994f9ca0
Author: Mat Kelly <machawk1@gmail.com>
Date:   Thu Dec 7 22:00:37 2017 -0500

    Fix typo

C:\dnload\ipfs-etc\ipwb>git diff
diff --git a/ipwb/__init__.py b/ipwb/__init__.py
index d41d25b..44a3f8b 100644
--- a/ipwb/__init__.py
+++ b/ipwb/__init__.py
@@ -1 +1 @@
-__version__ = '0.2017.12.07.2106'
+__version__ = '0.2017.12.07.2107'
diff --git a/ipwb/util.py b/ipwb/util.py
index 7527927..b26863c 100644
--- a/ipwb/util.py
+++ b/ipwb/util.py
@@ -40,7 +40,7 @@ def isDaemonAlive(hostAndPort="{0}:{1}".format(IPFSAPI_IP, IPFSAPI_PORT)):

     try:
         # OSError if ipfs not installed
-        subprocess.call(['ipfs', '--version'], stdout=open(devnull, 'wb'))
+        # subprocess.call(['ipfs', '--version'], stdout=open(devnull, 'wb'))

         # ConnectionError/AttributeError if IPFS daemon not running
         client.id()

Seems still the same situation:

C:\dnload\ipfs-etc>\python27\python.exe ipwb/ipwb replay
This version of ipwb is outdated. Please run pip install --upgrade ipwb.
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
 * Running on http://localhost:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [08/Dec/2017 20:08:15] "GET /webui/favicons/apple-touch-icon.png HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:15] "GET /webui/favicons/favicon-16x16.png HTTP/1.1" 200 -
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
127.0.0.1 - - [08/Dec/2017 20:08:22] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:22] "GET /webui/webui.css HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:22] "GET /webui/webui.js HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:22] "GET /webui/logo.png HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:22] "GET /daemon/status HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:22] "GET /config/openEndedPlaceHolder HTTP/1.1" 200 -
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
127.0.0.1 - - [08/Dec/2017 20:08:22] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:23] "GET /webui/webui.css HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:23] "GET /webui/webui.js HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:23] "GET /webui/daemonController.js HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:23] "GET /webui/favicons/apple-touch-icon.png HTTP/1.1" 200 -
127.0.0.1 - - [08/Dec/2017 20:08:23] "GET /webui/favicons/favicon-16x16.png HTTP/1.1" 200 -
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 49275)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 398, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 532, in show_uri
    sys.exit()
SystemExit

[machawk1]

@akavel A bit was done with this. Please do a fresh pull and let me know if you continue to experience the above issue.

[akavel]

Still not OK for me, though somewhat different symptoms now: after clicking the link on welcome page, I'm taken to the below URL:

http://localhost:5000/20160305192247/cs.odu.edu/~salam

and see the following logs:

C:\dnload\ipfs-etc>\python27\python.exe ipwb/ipwb replay
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
IPWB replay started on http://localhost:5000
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
'module' object has no attribute 'exec_info'

My current revision is:

C:\dnload\ipfs-etc\ipwb>git log -1 HEAD
commit b8f8fd4ae3ff5163ea8dd69fc0dff6b21182e88f
Merge: 1a9a518 24e3083
Author: Mat Kelly <machawk1@gmail.com>
Date:   Sat Dec 9 08:46:12 2017 -0500

    Merge pull request #322 from oduwsdl/issue-321

    Better describe replay script, fix README to align. Closes #321.

[machawk1]

@akavel Somehow exc_info had turned into exec_info at some point. This is the reason for the vague, uninformative error message you received. I fixed this in #323 thanks to your report and have pushed the changes to master. Can you re-pull from master and replicate the above so we can figure out what sort of exception is occurring?

[akavel]

Sure!

C:\dnload\ipfs-etc\ipwb>git log -1 HEAD
commit 866d7b484c450d43ac49fa586b6ee158ab1715a2
Merge: b8f8fd4 1947434
Author: Mat Kelly <machawk1@gmail.com>
Date:   Sat Dec 9 10:30:00 2017 -0500

    Merge pull request #324 from oduwsdl/issue-323-new

    Fixes syntax issue with reporting generic exceptions

Logs:

C:\dnload\ipfs-etc>\python27\python.exe ipwb/ipwb replay
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
IPWB replay started on http://localhost:5000
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
Retrieving URI-Ms from ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /ipwb/ipwb\samples\indexes\sample-encrypted.cdxj
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56133)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56135)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56137)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
----------------------------------------
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 56139)
Traceback (most recent call last):
  File "C:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "C:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "C:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "C:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "ipwb/ipwb\replay.py", line 399, in showMementoAtDatetime
    return show_uri(urir, datetime)
  File "ipwb/ipwb\replay.py", line 534, in show_uri
    sys.exit()
SystemExit
[...]

and more of similar-looking traces.

[akavel]

I modified replay.py l.533 to:

    print(sys.exc_info())

and got:

(<type 'exceptions.AttributeError'>, AttributeError("'module' object has no attribute 'SIGALRM'",), <traceback object at 0x0000000004A3C548>)

[akavel]

Here's a diff which fixes this for me:

diff --git a/ipwb/indexer.py b/ipwb/indexer.py
index d2bfc78..da2cb61 100755
--- a/ipwb/indexer.py
+++ b/ipwb/indexer.py
@@ -33,7 +33,7 @@ from __init__ import __version__ as ipwbVersion

 DEBUG = False

-IPFS_API = ipfsapi.Client(IPFSAPI_IP, IPFSAPI_PORT)
+IPFS_API = ipfsapi.Client(IPFSAPI_IP, IPFSAPI_PORT, timeout=10)

 # TODO: put this method definition below indexFileAt()
diff --git a/ipwb/replay.py b/ipwb/replay.py
index cc45140..71bf34d 100755
--- a/ipwb/replay.py
+++ b/ipwb/replay.py
@@ -507,13 +507,13 @@ def show_uri(path, datetime=None):
         def handler(signum, frame):
             raise HashNotFoundError()

-        signal.signal(signal.SIGALRM, handler)
-        signal.alarm(10)
+        #signal.signal(signal.SIGALRM, handler)
+        #signal.alarm(10)

         payload = IPFS_API.cat(digests[-1])
         header = IPFS_API.cat(digests[-2])

-        signal.alarm(0)
+        #signal.alarm(0)

     except ipfsapi.exceptions.TimeoutError:
         print("{0} not found at {1}".format(cdxjParts[0], digests[-1]))
@@ -530,7 +530,7 @@ def show_uri(path, datetime=None):
         return '', 404
     except:
         print('Unknown exception occurred while fetching from ipfs.')
-        print(sys.exc_info()[0])
+        print(sys.exc_info())
         sys.exit()

     if 'encryption_method' in jObj:
diff --git a/ipwb/util.py b/ipwb/util.py
index e92b2c9..df50b53 100644
--- a/ipwb/util.py
+++ b/ipwb/util.py
@@ -137,7 +137,7 @@ def retrieveMemCount():

 def datetimeToRFC1123(digits14):
     try:
-        locale.setlocale(locale.LC_TIME, 'en_US')
+        locale.setlocale(locale.LC_TIME, 'english') # for Windows (see: https://stackoverflow.com/questions/19709026/how-can-i-list-all-available-windows-locales-in-python-console#comment47807449_26927374)
     except locale.Error as e:
         locale.setlocale(locale.LC_TIME, 'en_US.utf8')
     d = datetime.datetime.strptime(digits14, '%Y%m%d%H%M%S')

[machawk1]

@akavel I originally was using the timeout with ipfsapi.Client but ran into some issues that caused the move to SIGALARM. I'll have to verify that reverting to a timeout does not resurface an old bug.

The locale issue should be handled more elegantly. I did not even considered Windows not supporting either of en_US or en_US.utf8. On macOS, if I swap out the line you provided, I get locale.Error: unsupported locale setting. The implicit case here (a result of me developing this on Mac initially) is for the try block to handle en_US (a macOS locale) and fall-through to en_US.utf8 as is often found on Linux. A third condition may need to be considered in Windows' english locale.

Thanks for your work looking into this bug so far!

[akavel]

Hm, actually I now see I mistakenly added the timeout in indexer.py; the timeout doesn't seem to work OK. Not sure how to properly inject it into requests via py-ipfs-api. Still, after removing the timeout, stuff finally starts to work for me.

[machawk1]

@akavel At one time we had it as a second argument on the cat() function on the ipfsapi.client instance. That was the aforementioned problematic issue I referenced that led me to change it to use signal instead.

[machawk1]

Re-tested this in a Windows 10 VM with cb4b5bc3fffa033a1b463ae71847f4acc765aa1e

C:\Users\Mat\Downloads\ipwb>ipwb replay
Executing first-run procedure on provided sample data.
Processing WARC records in salam-home.warc complete
IPWB replay started on http://localhost:5000
Retrieving URI-Ms from c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
Retrieving URI-Ms from c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
getting index file at /c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
Getting CDXJ Lines with the URI-R cs.odu.edu/~salam from samples/indexes/sample-encrypted.cdxj
Getting CDXJ Lines with cs.odu.edu/~salam in c:\python27\lib\site-packages\ipwb\samples\indexes\sample-encrypted.cdxj
CDXJ Line: edu,odu,cs)/~salam 20160305192247 {"locator": "urn:ipfs/QmeVWGtnfuJ1QnpmtKKnyArVgEpq7v31kktEfh6c8mDiXE/QmZWKQRBNXNrVZ69LoGpMNJi5NU66gDhnGtQukWJepv7Kr", "encryption_method": "xor", "encryption_key": "radon", "mime_type": "text/html", "status_code": "200"}
Unknown exception occurred while fetching from ipfs.
<type 'exceptions.AttributeError'>
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 53156)
Traceback (most recent call last):
  File "c:\python27\lib\SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "c:\python27\lib\SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "c:\python27\lib\SocketServer.py", line 652, in __init__
    self.handle()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 290, in handle
    rv = BaseHTTPRequestHandler.handle(self)
  File "c:\python27\lib\BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 325, in handle_one_request
    return self.run_wsgi()
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 267, in run_wsgi
    execute(self.server.app)
  File "c:\python27\lib\site-packages\werkzeug\serving.py", line 255, in execute
    application_iter = app(environ, start_response)
  File "c:\python27\lib\site-packages\flask\app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "c:\python27\lib\site-packages\flask\app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "c:\python27\lib\site-packages\flask\app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "c:\python27\lib\site-packages\ipwb\replay.py", line 218, in showMemento
    return show_uri(uri, newDatetime)
  File "c:\python27\lib\site-packages\ipwb\replay.py", line 593, in show_uri
    sys.exit()
SystemExit
----------------------------------------

[machawk1]

Again, the issue seems to stem from signal.signal(signal.SIGALRM, handler)

On Windows, running:

import signal
signal.signal(signal.SIGALRM, handler)

as pulled from the signal module docs produces an error:

AttributeError: 'module' object has no attribute 'SIGALRM'

From the very same docs: On Windows, signal() can only be called with SIGABRT, SIGFPE, SIGILL, SIGINT, SIGSEGV, or SIGTERM. A ValueError will be raised in any other case.

See https://stackoverflow.com/questions/8420422/python-windows-equivalent-of-sigalrm for a potential solution to replicate SIGALRM, though perhaps there is a simpler approach to accomplish what we are trying to do here on Windows.

What we are trying to do with signal is to set a timer for the two IPFS.cat() commands to complete. If they are not completed (and the signal is canceled), a timeout exception is raised.

[machawk1]

An attempt was made to move to the stopit module and remove the signal dependency in https://github.com/oduwsdl/ipwb/commit/7c3076b31d90789a0a860728e866a195a5e07bdd but the timeout is never executed per the stopit documentation.

Perhaps a threading-based solution with a Timer would work better but the calls are currently in a god function, so some refactoring should be done to isolate these commands and any commands dependent on the cat() results.

[akavel]

Could you possibly add some kind of a "temporary workaround" for Windows users, say, an option like --workaround-310-for-windows, disabling the timeout completely? As of now, this makes a world of a difference for a Windows user, i.e. uncommenting the timeout makes IPWB already a useful tool on Windows, while keeping it makes IPWB completely unusable...

[machawk1]

@akavel Thanks for the continued feedback. I am working on regaining a Windows test environment but for the time being, we may just need conditional functionality within the code until this fix is properly implemented.

I assure you that this fix to make ipwb useable for you on Windows is on my very-near radar.

[ibnesayeed]

I think the thread approach to fetch the two pieces of data from IPFS and then synchronize those threads would eliminate the need of this alarm. However, in the interim, we can try feature-detect mechanism (potentially by exception handling) to check the presence of SIGALARM and silently ignore it if it is not present.

[machawk1]

@akavel I have added a quick-fix to prevent the SIGNAL module commands from being executed on Windows. The change has been applied to the current master branch and via the latest release. Please give it a try and report back.

[machawk1]

As a followup on @ibnesayeed's suggestion, signal.ALRM on Windows produces an AttributeError. On Mac (and I presume Linux, untested), signal.SIGALRM returns an integer.

Similarly, signal.SIGABRT produces an integer on both Windows and Mac, so we may be able to try-catch on the call to signal.SIGALRM instead of explicitly conditioning on the Windows OS.

This check ought to be moot once we have the threading working correctly to re-enable the timeout functionality for Windows, but I wanted to follow-up on the previous message with the suboptimal OS-check solution.