commonsearch / cosr-back

Backend of Common Search. Analyses webpages and sends them to the index.
https://about.commonsearch.org
Apache License 2.0
122 stars 24 forks source link

Questions on deployment #44

Closed IvRRimum closed 8 years ago

IvRRimum commented 8 years ago

Hello,

Thanks for everyone who replied! Really helped me.

So all last week i have been trying to setup up this project using AWS.

I have multiple questions about development!

1) Does cosr-ops script provide everything needed for creation of Es clusters and all that worker stuff ?

2) You didnt mention you use rocksdb and gombo, i installed it manually in spark-master.

3) When i try to index test data, by running command make aws_spark_deploy_cosrback, it gives me error:

This happens after ALEXATOP1M is downloaded and it tried to write it to rocksdb.

   Traceback (most recent call last):
      File "urlserver/import.py", line 21, in <module>
        ds.import_dump()
      File "./urlserver/datasources/__init__.py", line 62, in import_dump
        for i, row in self.iter_dump():
      File "./urlserver/datasources/__init__.py", line 102, in iter_dump
        f = self.open_dump()
      File "./urlserver/datasources/__init__.py", line 144, in open_dump
        return GzipStreamFile(f)
      File "/cosr/back/venv/src/gzipstream/gzipstream/gzipstreamfile.py", line 62, in __init__
        super(GzipStreamFile, self).__init__(self._gzipstream)
      File "/usr/lib64/python2.6/io.py", line 921, in __init__
        raw._checkReadable()
    AttributeError: '_GzipStreamFile' object has no attribute '_checkReadable'

Any ideas ? I have been beating my head on this for a long time.

4) After running make aws_elasticsearch_create and successfully creating ES instances(3 of them). How do i access them ? Whats the port of ES to point frontend to ?

I really hope to finally set everything up, so i can work on the ISSUES( frontend and backend ). I have hight hopes for this project.

IvRRimum commented 8 years ago

Problem was that the file was not found, lack of documentation. closing