search

src-d / gitcollector

GNU General Public License v3.0
15 stars 14 forks source link

Progress export works incorrectly #35

Closed smacker closed 5 years ago

smacker commented 5 years ago

Queries executed one after another (first gitbase):

Screenshot 2019-06-18 at 14 27 18 Screenshot 2019-06-18 at 14 27 27

Logs:

$ docker logs -f srcd-tuxvbknvzgu_gitcollector_1
time="2019-06-18T12:23:13.5011812Z" level=debug msg="temporal dir: /tmp/gitcollector-downloader309492289" source="subcmd/download.go:59"
time="2019-06-18T12:23:13.5014106Z" level=debug msg="acces token found" source="subcmd/download.go:70"
time="2019-06-18T12:23:13.5017067Z" level=debug msg="allow updates on downloads: true" source="subcmd/download.go:80"
time="2019-06-18T12:23:13.5280681Z" level=debug msg="number of workers in the pool 2" source="subcmd/download.go:114"
time="2019-06-18T12:23:13.5283908Z" level=debug msg="worker pool is running" source="subcmd/download.go:117"
time="2019-06-18T12:23:13.5287481Z" level=debug msg="github provider started" source="subcmd/download.go:127"
time="2019-06-18T12:23:14.5927277Z" level=info msg=started id=0b17df5c-e47a-4c54-9ca6-33ece8fa658b job=download source="downloader/download.go:80" url="https://github.com/MLonCode/memlayout"
time="2019-06-18T12:23:14.6009875Z" level=info msg=started id=8772b554-d4f9-446e-8463-b3ffdf55092c job=download source="downloader/download.go:80" url="https://github.com/MLonCode/sonic"
time="2019-06-18T12:23:17.1232431Z" level=debug msg=cloned elapsed=2.5220538s id=8772b554-d4f9-446e-8463-b3ffdf55092c job=download source="downloader/download.go:121" url="https://github.com/MLonCode/sonic"
time="2019-06-18T12:23:17.1249695Z" level=debug msg="head commit found" head=76ed928082383174bd469ece51352dd17ac98a00 id=8772b554-d4f9-446e-8463-b3ffdf55092c job=download source="downloader/download.go:136" url="https://github.com/MLonCode/sonic"
time="2019-06-18T12:23:17.1265827Z" level=debug msg="root commit found" elapsed=1.5009ms id=8772b554-d4f9-446e-8463-b3ffdf55092c job=download root=15e56329858f20e662085ab51b9e3872e9589b03 source="downloader/download.go:148" url="https://github.com/MLonCode/sonic"
time="2019-06-18T12:23:17.1509094Z" level=debug msg=copied elapsed=16.4413ms id=8772b554-d4f9-446e-8463-b3ffdf55092c job=download source="downloader/download.go:183" url="https://github.com/MLonCode/sonic"
time="2019-06-18T12:23:17.6263435Z" level=debug msg=cloned elapsed=3.0334211s id=0b17df5c-e47a-4c54-9ca6-33ece8fa658b job=download source="downloader/download.go:121" url="https://github.com/MLonCode/memlayout"
time="2019-06-18T12:23:17.6277775Z" level=debug msg="head commit found" head=8f85636900147b9550e64beb1e05a7e532cbd67c id=0b17df5c-e47a-4c54-9ca6-33ece8fa658b job=download source="downloader/download.go:136" url="https://github.com/MLonCode/memlayout"
time="2019-06-18T12:23:17.6307986Z" level=debug msg="root commit found" elapsed=2.8949ms id=0b17df5c-e47a-4c54-9ca6-33ece8fa658b job=download root=a9796d6f1577e59f55eba5ce1df8f769f1fdc9ff source="downloader/download.go:148" url="https://github.com/MLonCode/memlayout"
time="2019-06-18T12:23:17.6480528Z" level=debug msg=copied elapsed=16.5896ms id=0b17df5c-e47a-4c54-9ca6-33ece8fa658b job=download source="downloader/download.go:183" url="https://github.com/MLonCode/memlayout"
time="2019-06-18T12:23:18.7432751Z" level=debug msg=fetched elapsed=1.093811s id=0b17df5c-e47a-4c54-9ca6-33ece8fa658b job=download source="downloader/download.go:217" url="https://github.com/MLonCode/memlayout"
time="2019-06-18T12:23:18.7507575Z" level=debug msg=commited elapsed=7.173ms id=0b17df5c-e47a-4c54-9ca6-33ece8fa658b job=download source="downloader/download.go:225" url="https://github.com/MLonCode/memlayout"
time="2019-06-18T12:23:18.7542478Z" level=info msg=finished elapsed=4.1613843s id=0b17df5c-e47a-4c54-9ca6-33ece8fa658b job=download source="downloader/download.go:95" url="https://github.com/MLonCode/memlayout"
time="2019-06-18T12:23:18.7555873Z" level=info msg=started id=bdb59e73-d630-4bfc-8557-086b5f164408 job=download source="downloader/download.go:80" url="https://github.com/MLonCode/evop"
time="2019-06-18T12:23:19.0736851Z" level=debug msg=fetched elapsed=1.9188181s id=8772b554-d4f9-446e-8463-b3ffdf55092c job=download source="downloader/download.go:217" url="https://github.com/MLonCode/sonic"
time="2019-06-18T12:23:19.0802461Z" level=debug msg=commited elapsed=6.1763ms id=8772b554-d4f9-446e-8463-b3ffdf55092c job=download source="downloader/download.go:225" url="https://github.com/MLonCode/sonic"
time="2019-06-18T12:23:19.092771Z" level=info msg=finished elapsed=4.4916263s id=8772b554-d4f9-446e-8463-b3ffdf55092c job=download source="downloader/download.go:95" url="https://github.com/MLonCode/sonic"
time="2019-06-18T12:23:19.0935319Z" level=info msg=started id=4739b6a6-e2da-4f01-9fab-19e0da252c45 job=download source="downloader/download.go:80" url="https://github.com/MLonCode/code2seq"
time="2019-06-18T12:23:19.6686057Z" level=debug msg=cloned elapsed=906.594ms id=bdb59e73-d630-4bfc-8557-086b5f164408 job=download source="downloader/download.go:121" url="https://github.com/MLonCode/evop"
time="2019-06-18T12:23:19.6696591Z" level=debug msg="head commit found" head=95dd16974e21c2076a583748107719a1830b8885 id=bdb59e73-d630-4bfc-8557-086b5f164408 job=download source="downloader/download.go:136" url="https://github.com/MLonCode/evop"
time="2019-06-18T12:23:19.6704667Z" level=debug msg="root commit found" elapsed="510.3µs" id=bdb59e73-d630-4bfc-8557-086b5f164408 job=download root=3855424c699a2262fea3d2aa8549cbfd57ea4da2 source="downloader/download.go:148" url="https://github.com/MLonCode/evop"
time="2019-06-18T12:23:19.6723609Z" level=debug msg=copied elapsed=1.2275ms id=bdb59e73-d630-4bfc-8557-086b5f164408 job=download source="downloader/download.go:183" url="https://github.com/MLonCode/evop"
time="2019-06-18T12:23:20.1712836Z" level=debug msg=fetched elapsed=498.4842ms id=bdb59e73-d630-4bfc-8557-086b5f164408 job=download source="downloader/download.go:217" url="https://github.com/MLonCode/evop"
time="2019-06-18T12:23:20.175682Z" level=debug msg=commited elapsed=3.1023ms id=bdb59e73-d630-4bfc-8557-086b5f164408 job=download source="downloader/download.go:225" url="https://github.com/MLonCode/evop"
time="2019-06-18T12:23:20.1781269Z" level=info msg=finished elapsed=1.422356s id=bdb59e73-d630-4bfc-8557-086b5f164408 job=download source="downloader/download.go:95" url="https://github.com/MLonCode/evop"
time="2019-06-18T12:23:20.1804704Z" level=info msg=started id=1a194f8e-b912-4329-aeb3-353175172ec2 job=download source="downloader/download.go:80" url="https://github.com/MLonCode/ncc"
time="2019-06-18T12:23:21.1734223Z" level=debug msg=cloned elapsed=2.0794445s id=4739b6a6-e2da-4f01-9fab-19e0da252c45 job=download source="downloader/download.go:121" url="https://github.com/MLonCode/code2seq"
time="2019-06-18T12:23:21.1739694Z" level=debug msg="head commit found" head=f6fb85ea05ad96535cc6cc087f133830ab6b26c1 id=4739b6a6-e2da-4f01-9fab-19e0da252c45 job=download source="downloader/download.go:136" url="https://github.com/MLonCode/code2seq"
time="2019-06-18T12:23:21.1745408Z" level=debug msg="root commit found" elapsed="338.9µs" id=4739b6a6-e2da-4f01-9fab-19e0da252c45 job=download root=f22be3e739b784c0daa27d3d4aac022a0459a14e source="downloader/download.go:148" url="https://github.com/MLonCode/code2seq"
time="2019-06-18T12:23:21.1799215Z" level=debug msg=copied elapsed=4.9723ms id=4739b6a6-e2da-4f01-9fab-19e0da252c45 job=download source="downloader/download.go:183" url="https://github.com/MLonCode/code2seq"
time="2019-06-18T12:23:23.1977548Z" level=debug msg=cloned elapsed=3.0166444s id=1a194f8e-b912-4329-aeb3-353175172ec2 job=download source="downloader/download.go:121" url="https://github.com/MLonCode/ncc"
time="2019-06-18T12:23:23.1982726Z" level=debug msg="head commit found" head=4e9eaeb6e7d3687cc73e73ae0fdf513f4e429e83 id=1a194f8e-b912-4329-aeb3-353175172ec2 job=download source="downloader/download.go:136" url="https://github.com/MLonCode/ncc"
time="2019-06-18T12:23:23.199415Z" level=debug msg="root commit found" elapsed=1.0163ms id=1a194f8e-b912-4329-aeb3-353175172ec2 job=download root=fc688735a2a79acfb3365f4423878c8fd9536c30 source="downloader/download.go:148" url="https://github.com/MLonCode/ncc"
time="2019-06-18T12:23:23.2109243Z" level=debug msg=copied elapsed=11.0515ms id=1a194f8e-b912-4329-aeb3-353175172ec2 job=download source="downloader/download.go:183" url="https://github.com/MLonCode/ncc"
time="2019-06-18T12:23:23.6000845Z" level=debug msg=fetched elapsed=388.3955ms id=1a194f8e-b912-4329-aeb3-353175172ec2 job=download source="downloader/download.go:217" url="https://github.com/MLonCode/ncc"
time="2019-06-18T12:23:23.6152337Z" level=debug msg=commited elapsed=14.7819ms id=1a194f8e-b912-4329-aeb3-353175172ec2 job=download source="downloader/download.go:225" url="https://github.com/MLonCode/ncc"
time="2019-06-18T12:23:23.6190663Z" level=info msg=finished elapsed=3.437998s id=1a194f8e-b912-4329-aeb3-353175172ec2 job=download source="downloader/download.go:95" url="https://github.com/MLonCode/ncc"
time="2019-06-18T12:23:23.6199417Z" level=info msg=started id=5faf236b-7043-4b59-9eb5-5cbed55474bb job=download source="downloader/download.go:80" url="https://github.com/MLonCode/ggnn_graph_classification"
time="2019-06-18T12:23:23.7704598Z" level=debug msg=fetched elapsed=2.5900296s id=4739b6a6-e2da-4f01-9fab-19e0da252c45 job=download source="downloader/download.go:217" url="https://github.com/MLonCode/code2seq"
time="2019-06-18T12:23:23.7784882Z" level=debug msg=commited elapsed=7.8475ms id=4739b6a6-e2da-4f01-9fab-19e0da252c45 job=download source="downloader/download.go:225" url="https://github.com/MLonCode/code2seq"
time="2019-06-18T12:23:23.780564Z" level=info msg=finished elapsed=4.6866314s id=4739b6a6-e2da-4f01-9fab-19e0da252c45 job=download source="downloader/download.go:95" url="https://github.com/MLonCode/code2seq"
time="2019-06-18T12:23:23.7816651Z" level=info msg=started id=82d7765e-6ad8-4115-86e5-db6a81610158 job=download source="downloader/download.go:80" url="https://github.com/MLonCode/astminer"
time="2019-06-18T12:23:25.5383275Z" level=debug msg=cloned elapsed=1.7562361s id=82d7765e-6ad8-4115-86e5-db6a81610158 job=download source="downloader/download.go:121" url="https://github.com/MLonCode/astminer"
time="2019-06-18T12:23:25.5398515Z" level=debug msg="head commit found" head=d303f59581a4b42e39938677441d4a75ad05313e id=82d7765e-6ad8-4115-86e5-db6a81610158 job=download source="downloader/download.go:136" url="https://github.com/MLonCode/astminer"
time="2019-06-18T12:23:25.542616Z" level=debug msg="root commit found" elapsed=2.6189ms id=82d7765e-6ad8-4115-86e5-db6a81610158 job=download root=cb50054904c2f63654d65bf1d1f472efdb7ed5cf source="downloader/download.go:148" url="https://github.com/MLonCode/astminer"
time="2019-06-18T12:23:25.5446253Z" level=debug msg=copied elapsed=1.4631ms id=82d7765e-6ad8-4115-86e5-db6a81610158 job=download source="downloader/download.go:183" url="https://github.com/MLonCode/astminer"
time="2019-06-18T12:23:26.6332225Z" level=debug msg=fetched elapsed=1.0879254s id=82d7765e-6ad8-4115-86e5-db6a81610158 job=download source="downloader/download.go:217" url="https://github.com/MLonCode/astminer"
time="2019-06-18T12:23:26.6345237Z" level=debug msg=commited elapsed=1.1933ms id=82d7765e-6ad8-4115-86e5-db6a81610158 job=download source="downloader/download.go:225" url="https://github.com/MLonCode/astminer"
time="2019-06-18T12:23:26.6354998Z" level=info msg=finished elapsed=2.8535362s id=82d7765e-6ad8-4115-86e5-db6a81610158 job=download source="downloader/download.go:95" url="https://github.com/MLonCode/astminer"
time="2019-06-18T12:23:26.6365526Z" level=info msg=started id=bf3cfe8b-2597-42f8-8f64-b8dc2c7146d9 job=download source="downloader/download.go:80" url="https://github.com/MLonCode/structured-neural-summarization"
time="2019-06-18T12:23:27.7212567Z" level=debug msg=cloned elapsed=1.1199898s id=bf3cfe8b-2597-42f8-8f64-b8dc2c7146d9 job=download source="downloader/download.go:121" url="https://github.com/MLonCode/structured-neural-summarization"
time="2019-06-18T12:23:27.721759Z" level=debug msg="head commit found" head=9b0535ce760ad7af293b0de5b82c4070d900b915 id=bf3cfe8b-2597-42f8-8f64-b8dc2c7146d9 job=download source="downloader/download.go:136" url="https://github.com/MLonCode/structured-neural-summarization"
time="2019-06-18T12:23:27.7220215Z" level=debug msg="root commit found" elapsed="8.9µs" id=bf3cfe8b-2597-42f8-8f64-b8dc2c7146d9 job=download root=9b0535ce760ad7af293b0de5b82c4070d900b915 source="downloader/download.go:148" url="https://github.com/MLonCode/structured-neural-summarization"
time="2019-06-18T12:23:27.7401005Z" level=debug msg=copied elapsed=1.2315ms id=bf3cfe8b-2597-42f8-8f64-b8dc2c7146d9 job=download source="downloader/download.go:183" url="https://github.com/MLonCode/structured-neural-summarization"
time="2019-06-18T12:23:28.7461138Z" level=debug msg=fetched elapsed=1.005425s id=bf3cfe8b-2597-42f8-8f64-b8dc2c7146d9 job=download source="downloader/download.go:217" url="https://github.com/MLonCode/structured-neural-summarization"
time="2019-06-18T12:23:28.7500071Z" level=debug msg=commited elapsed=2.8174ms id=bf3cfe8b-2597-42f8-8f64-b8dc2c7146d9 job=download source="downloader/download.go:225" url="https://github.com/MLonCode/structured-neural-summarization"
time="2019-06-18T12:23:28.7547999Z" level=info msg=finished elapsed=2.153722s id=bf3cfe8b-2597-42f8-8f64-b8dc2c7146d9 job=download source="downloader/download.go:95" url="https://github.com/MLonCode/structured-neural-summarization"
time="2019-06-18T12:23:33.7560767Z" level=debug msg="waiting new metrics" metrics=library source="metrics/metrics.go:138"
... many repeated "waiting new metrics" ...
smacker commented 5 years ago

numbers don't change after a very long time when all repos must be downloaded already for sure.

smacker commented 5 years ago

hm. no. it finally updated. full log: https://gist.github.com/smacker/f34da47c01ebd536bb12cce04b91b14e does it mean it was cloning for 20 minutes? Still there was like 10 minutes difference while I saw mismatch of data in gitbase and in postgres.