'collection for git: starting' repeatedly appear

[Julianbaozi]

It can never be finished. the log: all.log

[valeriocos]

In fact I can run 'debug' but not 'run'

Can you give more details about this? Thanks!

[junliangyu96]

elasticsearch==6.3.1 elasticsearch-dsl==6.3.1 requests==2.21.0 urllib3==1.24.3 PyMySQL==0.7.0 geopy>=1.21.0 pandas==0.25.3 statsmodels >= 0.11.1

This is the packages I'm using from the setting: Python Interpreter There is always an alert that says package requirement 'grimoirelab-elk' is not satisfied. Yet I click install requirement nothing happens.

This time when I debug, message:

/usr/bin/python3.6 /snap/pycharm-community/192/plugins/python-ce/helpers/pydev/pydevd.py --multiproc --qt-support=auto --client 127.0.0.1 --port 41665 --file micro.py -raw --enrich --cfg ./setup.cfg --backends git pydev debugger: process 14184 is connecting Connected to pydev debugger (build 201.6668.115) Traceback (most recent call last): File "", line 971, in _find_and_load File "", line 955, in _find_and_load_unlocked File "", line 665, in _load_unlocked File "", line 678, in exec_module File "", line 219, in _call_with_frames_removed File "/home/yjl/sources/grimoirelab-sirmordred/sirmordred/config.py", line 26, in from sirmordred.task import Task File "/home/yjl/sources/grimoirelab-sirmordred/sirmordred/task.py", line 28, in from grimoire_elk.elk import get_ocean_backend File "/home/yjl/sources/grimoirelab-elk/grimoire_elk/elk.py", line 38, in from .utils import get_elastic File "/home/yjl/sources/grimoirelab-elk/grimoire_elk/utils.py", line 78, in from .enriched.askbot import AskbotEnrich File "/home/yjl/sources/grimoirelab-elk/grimoire_elk/enriched/askbot.py", line 29, in from .enrich import Enrich, metadata File "/home/yjl/sources/grimoirelab-elk/grimoire_elk/enriched/enrich.py", line 43, in from .study_ceres_onion import ESOnionConnector, onion_study File "/home/yjl/sources/grimoirelab-elk/grimoire_elk/enriched/study_ceres_onion.py", line 25, in import pandas File "/home/yjl/.local/lib/python3.6/site-packages/pandas/init.py", line 55, in from pandas.core.api import ( File "/home/yjl/.local/lib/python3.6/site-packages/pandas/core/api.py", line 5, in from pandas.core.arrays.integer import ( File "/home/yjl/.local/lib/python3.6/site-packages/pandas/core/arrays/init.py", line 13, in from .sparse import SparseArray # noqa: F401 File "/home/yjl/.local/lib/python3.6/site-packages/pandas/core/arrays/sparse/init.py", line 3, in from pandas.core.arrays.sparse.accessor import SparseAccessor, SparseFrameAccessor File "/home/yjl/.local/lib/python3.6/site-packages/pandas/core/arrays/sparse/accessor.py", line 10, in from pandas.core.arrays.sparse.array import SparseArray File "/home/yjl/.local/lib/python3.6/site-packages/pandas/core/arrays/sparse/array.py", line 46, in from pandas.core.indexers import check_array_indexer ImportError: cannot import name 'check_array_indexer'

Thank you!

[junliangyu96]

I changed pandas to 0.24.2. It can debug now but when I run it's the same problem.

[yjl96]

After fetching many repos, how to set the micro only for enrichment? can you give me an example of both parameters passed in micro.py and the setup.py?

[valeriocos]

https://github.com/chaoss/grimoirelab-sirmordred/issues/462#issuecomment-623501039

Hi @yjl96, did you try to follow this getting started: https://github.com/chaoss/grimoirelab-sirmordred/blob/master/Getting-Started.md#setting-up-pycharm- ?

https://github.com/chaoss/grimoirelab-sirmordred/issues/462#issuecomment-623509303

That's a bit weird. I guess it's a configuration problem with your pycharm. Please follow the getting started at https://github.com/chaoss/grimoirelab-sirmordred/blob/master/Getting-Started.md#setting-up-pycharm- and see if the error is gone. Several users followed it, and nobody reported a problem similar to yours.

After fetching many repos, how to set the micro only for enrichment? can you give me an example of both parameters passed in micro.py and the setup.py?

Examples of micro-mordred are available at:

• https://github.com/chaoss/grimoirelab-sirmordred/blob/master/Getting-Started.md#execution-
• https://github.com/chaoss/grimoirelab-sirmordred#micro-mordred-

The setup.cfg is the same of mordred, an example of setup.cfg and projects.json is at:

• https://github.com/chaoss/grimoirelab-sirmordred/blob/master/utils/setup.cfg
• https://github.com/chaoss/grimoirelab-sirmordred/blob/master/utils/projects.json

[Julianbaozi]

Hi. I reinstalled the environment and it's working now. Thank you!

What is the order of computation if I pass more than one projects with git and github? Is it project1-git, project1-gituhb, project1-git_enrich, project1-github_enrich, project2?

[Julianbaozi]

If I use tokens from different account, I'm still using the same IP. Will I be banned because of this?

[valeriocos]

Hi @Julianbaozi !

Hi. I reinstalled the environment and it's working now. Thank you!

Great news! You're welcome

What is the order of computation if I pass more than one projects with git and github? Is it project1-git, project1-gituhb, project1-git_enrich, project1-github_enrich, project2?

The data sources' executions are independent, so there is no precise order. Each data source executes different steps (collection, enrichment, etc.)

If I use tokens from different account, I'm still using the same IP. Will I be banned because of this?

It depends on the number of tokens used, however in general it shouldn't be a problem. Are you using the tokens at the same time or sequentially?

[junliangyu96]

At the same time. Distributed on one machine

On Fri, May 8, 2020 at 9:33 AM valerio notifications@github.com wrote:

Hi @Julianbaozi https://github.com/Julianbaozi !

Hi. I reinstalled the environment and it's working now. Thank you!

Great news! You're welcome

What is the order of computation if I pass more than one projects with git and github? Is it project1-git, project1-gituhb, project1-git_enrich, project1-github_enrich, project2?

The data sources' executions are independent, so there is no precise order. Each data source executes different steps (collection, enrichment, etc.)

If I use tokens from different account, I'm still using the same IP. Will I be banned because of this?

It depends on the number of tokens used, however in general it shouldn't be a problem. Are you using the tokens at the same time or sequentially?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/grimoirelab-sirmordred/issues/462#issuecomment-625900222, or unsubscribe https://github.com/notifications/unsubscribe-auth/APNVJ7SLCMMIWLU42ECJAY3RQQX4DANCNFSM4MQUC5SQ .

[junliangyu96]

I used 4 tokens. The projects are paralleled, sharing the 4 tokens

On Fri, May 8, 2020 at 9:33 AM valerio notifications@github.com wrote:

Hi @Julianbaozi https://github.com/Julianbaozi !

Hi. I reinstalled the environment and it's working now. Thank you!

Great news! You're welcome

What is the order of computation if I pass more than one projects with git and github? Is it project1-git, project1-gituhb, project1-git_enrich, project1-github_enrich, project2?

The data sources' executions are independent, so there is no precise order. Each data source executes different steps (collection, enrichment, etc.)

If I use tokens from different account, I'm still using the same IP. Will I be banned because of this?

It depends on the number of tokens used, however in general it shouldn't be a problem. Are you using the tokens at the same time or sequentially?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/grimoirelab-sirmordred/issues/462#issuecomment-625900222, or unsubscribe https://github.com/notifications/unsubscribe-auth/APNVJ7SLCMMIWLU42ECJAY3RQQX4DANCNFSM4MQUC5SQ .

[valeriocos]

thank you for the details @Julianbaozi ! You shouldn't have any problem. Keep an eye on 429 http errors, these ones are generated when performing too many requests. After several warnings of this type, you might be banned.

[valeriocos]

@Julianbaozi can we close this issue?

[junliangyu96]

Yes. Thank you for your help!

On Mon, May 11, 2020 at 4:53 AM valerio notifications@github.com wrote:

@Julianbaozi https://github.com/Julianbaozi can we close this issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chaoss/grimoirelab-sirmordred/issues/462#issuecomment-626655461, or unsubscribe https://github.com/notifications/unsubscribe-auth/APNVJ7TV6PEODZPWBWT7BF3RQ7RK7ANCNFSM4MQUC5SQ .

[valeriocos]

Great, you're welcome!

[Julianbaozi]

Hi I ran into some new problems.

I want to run this on linux without UI, can I still use pycharm? Right now I'm trying just to download sources and built a virtual environment outside the folder sources (not in sirmordred). And an error occur:

Traceback (most recent call last): (pid=29063) File "/home/junliang/sources/grimoirelab-sirmordred/utils/micro.py", line 30, in (pid=29063) from sirmordred.config import Config (pid=29063) ModuleNotFoundError: No module named 'sirmordred' (pid=29064) Traceback (most recent call last): (pid=29064) File "/home/junliang/sources/grimoirelab-sirmordred/utils/micro.py", line 30, in (pid=29064) from sirmordred.config import Config (pid=29064) ModuleNotFoundError: No module named 'sirmordred'

I added those sources to sys.path. The new sys.path is

['/home/junliang/venvs/gl/lib/python3.6/site-packages/ray/thirdparty_files', '/home/junliang/venvs/gl/lib/python3.6/site-packages/ray/pickle5_files', '/home/junliang/gl', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/home/junliang/venvs/gl/lib/python3.6/site-packages', '/home/junliang/venvs/gl/src/grimoirelab-elk', '/home/junliang/venvs/gl/src/grimoirelab-cereslib', '/home/junliang/sources/grimoirelab-manuscripts', '/home/junliang/venvs/gl/src/grimoirelab-toolkit', '/home/junliang/venvs/gl/src/grimoirelab-sortinghat', '/home/junliang/venvs/gl/src/grimoirelab-kidash', '/home/junliang/venvs/gl/src/grimoirelab-graal', '/home/junliang/venvs/gl/src/grimoirelab-perceval', '/home/junliang/venvs/gl/src/grimoirelab-perceval-puppet', '/home/junliang/venvs/gl/src/grimoirelab-perceval-opnfv', '/home/junliang/venvs/gl/src/grimoirelab-perceval-mozilla', '/home/junliang/venvs/gl/src/grimoirelab-perceval-finos', '/home/junliang/venvs/gl/src/grimoirelab-sigils', '/home/junliang/sources/grimoirelab-cereslib', '/home/junliang/sources/grimoirelab-elk', '/home/junliang/sources/grimoirelab-graal', '/home/junliang/sources/grimoirelab-kidash', '/home/junliang/sources/grimoirelab-kingarthur', '/home/junliang/sources/grimoirelab-manuscripts', '/home/junliang/sources/grimoirelab-perceval', '/home/junliang/sources/grimoirelab-perceval-finos', '/home/junliang/sources/grimoirelab-perceval-mozilla', '/home/junliang/sources/grimoirelab-perceval-opnfv', '/home/junliang/sources/grimoirelab-perceval-puppet', '/home/junliang/sources/grimoirelab-sigils', '/home/junliang/sources/grimoirelab-sirmordred', '/home/junliang/sources/grimoirelab-sortinghat', '/home/junliang/sources/grimoirelab-toolkit']

sirmordred is in the path

So I don't know what to do to import those packages.

Thank you!

[valeriocos]

Hi @Julianbaozi , please consider to open a new issue.

The idea to create a virtual environment is a good way to go, however instead of adding the sources to sys.path, you should install Mordred with pip (using the code in your local copy). For instance:

1. $ python3 -m venv /tmp/test <-- create virtual env
2. $ source /tmp/test/bin/activate <-- activate it
3. $ cd .../mordred <- go to the local repo of Mordred
4. ../mordred$ pip install -r requirements.txt
5. ../mordred$ pip install .
6. $ sirmodred --help <- check that mordred has been installed

[Julianbaozi]

That worked thank you!