Closed chaconnewu closed 8 years ago
@chaconnewu sorry about that! there's a step missing in the docs:
./scripts/import_commoncrawl.sh 0
Can you confirm it fixes the issue?
Thanks @sylvinus! I confirm it fixes the issue. However, another error will generate after the job is running for a while:
Caught Python exception in generator! Traceback (most recent call last): File "/cosr/back/cosrlib/utils.py", line 26, in wrapped for x in fn(_args, *_kwargs): File "/cosr/back/jobs/spark/index.py", line 87, in iter_records warc_file = open_warc_file(filename, from_commoncrawl=(not args.warc_files)) File "cosrlib/webarchive.py", line 39, in open_warc_file pds = conn.get_bucket('aws-publicdatasets') File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 503, in get_bucket return self.head_bucket(bucket_name, headers=headers) File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 522, in head_bucket response = self.make_request('HEAD', bucket_name, headers=headers) File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 665, in make_request retry_handler=retry_handler File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request retry_handler=retry_handler) File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1030, in _mexe raise ex gaierror: [Errno -2] Name or service not known
I've pushed a tentative fix: https://github.com/commonsearch/cosr-back/commit/30f7aff9252a96dd34e7d2a8d5fdc5acd56687e4
Can you pull and see if it works?
Yes, it worked!
This is the output after indexing: 16/03/16 03:06:57 INFO PythonRunner: Times: total = 437675, boot = 144, init = 329, finish = 437202 16/03/16 03:06:57 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 118012 bytes result sent to driver 16/03/16 03:06:57 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 437930 ms on localhost (1/1) 16/03/16 03:06:57 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 16/03/16 03:06:57 INFO DAGScheduler: ResultStage 0 (count at /cosr/back/jobs/spark/index.py:165) finished in 437.994 s 16/03/16 03:06:57 INFO DAGScheduler: Job 0 finished: count at /cosr/back/jobs/spark/index.py:165, took 438.235379 s Indexed 695 WARC records
Awesome :)
When I run
spark-submit jobs/spark/index.py --warc_limit 1 --only_homepages --profile
as described in README.md, the follow error will appear:
16/03/15 07:15:18 INFO BlockManagerMaster: Registered BlockManager Traceback (most recent call last): File "/cosr/back/jobs/spark/index.py", line 174, in
spark_main()
File "/cosr/back/jobs/spark/index.py", line 145, in spark_main
warc_filenames = list_warc_filenames()
File "/cosr/back/jobs/spark/index.py", line 72, in list_warc_filenames
warc_files = list_commoncrawl_warc_filenames(limit=args.warc_limit, skip=args.warc_skip)
File "/cosr/back/cosrlib/webarchive.py", line 22, in list_commoncrawl_warc_filenames
with open(warc_paths, "r") as f:
IOError: [Errno 2] No such file or directory: '/cosr/back/local-data/common-crawl/warc.paths.txt'