localhost: 27017 connection refused

oimaasi commented 6 years ago

I installed the required packages according to the instruction, but got the following error when trying to run the script. Is there anything else I should setup beside the steps detailed in the installation guide? Thank you.

File "/usr/lib/python3.6/site-packages/pymongo/collection.py", line 1380, in create_index
    self.__create_index(keys, kwargs)
  File "/usr/lib/python3.6/site-packages/pymongo/collection.py", line 1286, in __create_index
    with self._socket_for_writes() as sock_info:
  File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.6/site-packages/pymongo/mongo_client.py", line 716, in _get_socket
    server = self._get_topology().select_server(selector)
  File "/usr/lib/python3.6/site-packages/pymongo/topology.py", line 142, in select_server
    address))
  File "/usr/lib/python3.6/site-packages/pymongo/topology.py", line 118, in select_servers
    self._error_message(selector))

pymongo.errors.ServerSelectionTimeoutError: localhost:27017:  [Errno 111] Connection refused

EDIT: Mongo has to be installed and started first. In Arch Linux the package is called mongodb. It's a little bit confusing for people who are not familiar with this field, because for instance if you install opencv-python, it will actually install opencv itself and its python binding, which is apparently not the case for pymongo.

Info about the platform I am running: Arch Linux + Python 3.6 + mongodb 3.6.3-1 + firefox 59.

oimaasi commented 6 years ago

Ok, I have to install mongodb and then start the service. Now when I run with dumb_site option, it will give the following error

Traceback (most recent call last):
  File "./yahoo-groups-backup.py", line 129, in <module>
    main()
  File "./yahoo-groups-backup.py", line 125, in main
    arguments, cfg_args)
  File "./yahoo-groups-backup.py", line 103, in invoke_subcommand
    return module.command(args)
  File "--/Packages/yahoo-groups-backup/yahoo_groups_backup/subcommands/dump_site.py", line 334, in command
    ds.run()
  File "--/Packages/yahoo-groups-backup/yahoo_groups_backup/subcommands/dump_site.py", line 317, in run
    self.render_config()
  File "--/Packages/yahoo-groups-backup/yahoo_groups_backup/subcommands/dump_site.py", line 216, in render_config
    'lastMessageTime': self.db.get_latest_message().get('postDate'),
  File "--/Packages/yahoo-groups-backup/yahoo_groups_backup/backup_db.py", line 91, in get_latest_message
    return next(self.yield_all_messages())
StopIteration

oimaasi commented 6 years ago

Tried scrape_messages, it started firefox, but seemed to be happening. Then after a while, it produced the following errors:

Traceback (most recent call last):
  File "./yahoo-groups-backup.py", line 129, in <module>
    main()
  File "./yahoo-groups-backup.py", line 125, in main
    arguments, cfg_args)
  File "./yahoo-groups-backup.py", line 103, in invoke_subcommand
    return module.command(args)
  File "--/Packages/yahoo-groups-backup/yahoo_groups_backup/subcommands/scrape_messages.py", line 32, in command
    arguments['--password'])
  File "--/Packages/yahoo-groups-backup/yahoo_groups_backup/scraper.py", line 29, in __init__
    self.br = splinter.Browser(driver)
  File "/usr/lib/python3.6/site-packages/splinter/browser.py", line 63, in Browser
    return driver(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/splinter/driver/webdriver/firefox.py", line 39, in __init__
    self.driver = Firefox(firefox_profile)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/webdriver.py", line 81, in __init__
    self.binary, timeout)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__
    self.binary.launch_browser(self.profile, timeout=timeout)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable(timeout=timeout)
  File "/usr/lib/python3.6/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 106, in _wait_until_connectable
    % (self.profile.path))
selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpr0umenb9 If you specified a log_file in the FirefoxBinary constructor, check it for details.

Exception ignored in: <bound method YahooBackupScraper.__del__ of <yahoo_groups_backup.scraper.YahooBackupScraper object at 0x7f8c0daf5978>>
Traceback (most recent call last):
  File "--/Packages/yahoo-groups-backup/yahoo_groups_backup/scraper.py", line 32, in __del__
    self.br.quit()
AttributeError: 'YahooBackupScraper' object has no attribute 'br'

oimaasi commented 6 years ago

Is there any specific requirement about the version of Mongo? Since other python packages were installed from requirement.txt, they have exactly the version as specified, so the remaining uncertainty could only be Mongo and Firefox?

brewingcode commented 6 years ago

Just in case anyone else finds this, I was seeing the exact same thing on macOS as https://github.com/csaftoiu/yahoo-groups-backup/issues/47#issuecomment-390253161: the scraper would open a Firefox window and just sit for 30 seconds or so, then fail out with selenium.common.exceptions.WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpr0umenb9 If you specified a log_file in the FirefoxBinary constructor, check it for details. .

I was able to quickly fix it by swapping to Chrome instead of Firefox:

$ brew tap homebrew/cask
$ brew cask install chromedriver
$ ./yahoo-groups-backup.py $COMMAND $GROUPNAME --driver=chrome
Detecting the log-in page...
Detecting the log-in page...
Detecting the log-in page...
Detecting the log-in page...
Detecting the log-in page...
Detecting the log-in page...
Inserted message #19471 by /[redacted]@...
Detecting the log-in page...
Detecting the log-in page...

Note: you install chromedriver but pass --driver=chrome to the Python script.

csaftoiu / yahoo-groups-backup

localhost: 27017 connection refused #47