Crawling fails with `exceptions.ImportError: cannot import name suppress` in docker-compose setup

[babajideowoyele]

On Ubuntu setup on a VirtualBox, I ran ´docker-compose up´ and I am getting these errors as displayed below. Can you help with the issue?

2021-04-23T17:44:15+0000 [Launcher,59/stderr] Unhandled error in Deferred:
crawler_1   | 2021-04-23T17:44:15+0000 [Launcher,59/stderr] 
crawler_1   |   Traceback (most recent call last):
crawler_1   |     File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 172, in crawl
crawler_1   |       return self._crawl(crawler, *args, **kwargs)
crawler_1   |     File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 176, in _crawl
crawler_1   |       d = crawler.crawl(*args, **kwargs)
crawler_1   |     File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1613, in unwindGenerator
crawler_1   |       return _cancellableInlineCallbacks(gen)
crawler_1   |     File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1529, in _cancellableInlineCallbacks
crawler_1   |       _inlineCallbacks(None, g, status)
crawler_1   |   --- <exception caught here> ---
crawler_1   |     File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
crawler_1   |       result = g.send(result)
crawler_1   |     File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 98, in crawl
crawler_1   |       six.reraise(*exc_info)
crawler_1   |     File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 82, in crawl
crawler_1   |       yield self.engine.open_spider(self.spider, start_requests)
crawler_1   |   exceptions.ImportError: cannot import name suppress

[boogheta]

Hello, how did you reach this point?

• Did you follow exactly the install documentation in the readme?
• Did you change anything to the configuration files?
• Did you do docker-compose pull or docker-compose build?

[DaniloOliveira28]

I am with the same erroe

[vidbina]

Hey @boogheta, @babajideowoyele and I followed the instructions from the README, copied the env files over, and ran docker-compose build on a clean machine (pull implied). In all cases, we ended up with this error.

Was wondering if it my have something to do with contextlib which doesn't include supress unless the contextlib2 compat package is used. The Python stdlib includes contextlib.supress since 3.4 but upgrading to this version would require some retrofitting on the codebase since the lang upgrade isn't backwards compatible (lambdas were breaking since they are arity 1 in Python 3 and I'm guessing more would break in the process of upgrading the lang). Before trying to figure out if we can install contextlib2, it would help to know what is different about the environments that are happily running out there in the wild. What are we missing.

Disclaimer: not involved with the project enough to know anything 🤷🏿, so I have no clue what I'm really talking about here. Was just trying to help getting something set up to run some tests.

[boogheta]

Hello all,

All right, I first thought your problem was regarding installation, but this is a runtime error of the crawler actually I was mistaken. I can actually reproduce it, so I will investigate and keep you posted.

But I don't think python3.4 gets into the issue as Hyphe unfortunately still runs full using python2.7 ;)

[boogheta]

All right, should be fixed with https://github.com/medialab/hyphe/commit/e59e87843da1c8b751b6f8f9f341116feb7061e4 Can you please git pull on the repository, run the build again, then start over and tell if this is fixed also on your side?

By the way, I'm excited to see that many unknown new users, may I ask out of curiosity and without any indiscretion how you all found out about Hyphe and what you intend to do with it? Thanks!

[babajideowoyele]

Thanks Benjamin.

David is mentoring me in data science for my research and so i always go to him whenever things i expect to use as "plug and play" for my research breakdown. Most times i am just to new to coding to do basic things. So the error you see comes from that context.

I am a daul PhD Candidate at the Hasso Plattner Institute and the Erasmus University. I started to look into hyperlink analysis tools since 2018. I used Voson ( ueberlink.com) until recently when the host researchers announced they would not be continuing with the project. I did one crawl of an Innovation network ( eit.europa.eu) but never got to obtain deeper outlinks and inlinks. I gave up on this approach until i stumbled upon Hyphe in a job description. They required the potential candidate to have a good knowledge of Hyphe and Gephi. That's when i discovered hyphe and watched the introductory video. I was then looking to cawl Hyperlinks with your tutorial but then the errors started mounting up.

Hopefully this can be a game changing tool for my work. The sustainability transitions community can benefit from using this approach to studying cultural phenomena.

Best regards

Babajide

Get Outlook for Androidhttps://aka.ms/ghei36

---

From: Benjamin Ooghe-Tabanou @.> Sent: Wednesday, April 28, 2021 4:42:07 PM To: medialab/hyphe @.> Cc: Babajide Owoyele @.>; Mention @.> Subject: Re: [medialab/hyphe] Crawling fails with exceptions.ImportError: cannot import name suppress in docker-compose setup (#407)

All right, should be fixed with e59e878https://github.com/medialab/hyphe/commit/e59e87843da1c8b751b6f8f9f341116feb7061e4 Can you please git pull on the repository, run the build again, then start over and tell if this is fixed also on your side?

By the way, I'm excited to see that many unknown new users, may I ask out of curiosity and without any indiscretion how you all found out about Hyphe and what you intend to do with it? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/medialab/hyphe/issues/407#issuecomment-828512363, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALO56KV7ENQ3ZWZNOKWNBDDTLAND7ANCNFSM43PCFKIA.

[boogheta]

Nice! Thanks a lot for this feedback Babajide, it's very interesting! May I know what the job offer was? It's quite cool to realize some people are hiring others using Hyphe as a required competence! :)

On a sidenode, did you get a chance to try the fix? May I close the issue?

[babajideowoyele]

Hi the job link is here https://www.phd-positions.dk/?p=207030.

I am getting this error after cloning. Docker is already installed " C:\WINDOWS\system32\hyphe> docker-compose pull Traceback (most recent call last): File "docker\api\client.py", line 214, in _retrieve_server_version File "docker\api\daemon.py", line 181, in version File "docker\utils\decorators.py", line 46, in inner File "docker\api\client.py", line 237, in _get File "requests\sessions.py", line 543, in get File "requests\sessions.py", line 530, in request File "requests\sessions.py", line 643, in send File "requests\adapters.py", line 439, in send File "urllib3\connectionpool.py", line 670, in urlopen File "urllib3\connectionpool.py", line 392, in _make_request File "http\client.py", line 1255, in request File "http\client.py", line 1301, in _send_request File "http\client.py", line 1250, in endheaders File "http\client.py", line 1010, in _send_output File "http\client.py", line 950, in send File "docker\transport\npipeconn.py", line 32, in connect File "docker\transport\npipesocket.py", line 23, in wrapped File "docker\transport\npipesocket.py", line 72, in connect File "docker\transport\npipesocket.py", line 52, in connect pywintypes.error: (2, 'CreateFile', 'The system cannot find the file specified.')

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "docker-compose", line 3, in File "compose\cli\main.py", line 81, in main File "compose\cli\main.py", line 199, in perform_command File "compose\cli\command.py", line 60, in project_from_options File "compose\cli\command.py", line 152, in get_project File "compose\cli\docker_client.py", line 41, in get_client File "compose\cli\docker_client.py", line 170, in docker_client File "docker\api\client.py", line 197, in init File "docker\api\client.py", line 221, in _retrieve_server_version docker.errors.DockerException: Error while fetching server API version: (2, 'CreateFile', 'The system cannot find the file specified.') [13284] Failed to execute script docker-compose PS C:\WINDOWS\system32\hyphe> docker compose build The new 'docker compose' command is currently experimental. To provide feedback or request new features please open issues at https://github.com/docker/compose-cli error during connect: In the default daemon configuration on Windows, the docker client must be run with elevated privileges to connect.: Get "http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.24/info": open //./pipe/docker_engine: The system cannot find the file specified. "

[babajideowoyele]

Dear Benjamin. It works. I just used another notebook. A Lenovo legion with a fresh install of everything. It works. Thanks

[boogheta]

Great! FYI, the problem you had on the other machine seems more like a Docker configuration one than a Hyphe one. It's quite common that Docker installs lack a few settings to enable any user to run commands, here it looks like your user misses these rights, you shuld be able to find solutions on the Docker for windows official documentation