search

jfalken / github_commit_crawler

Tool used to continuously monitor a Github org for mistaken public commits
165 stars 33 forks source link

UI Does Not Update with Results After GHCC Scans #9

Open thofli opened 8 years ago

thofli commented 8 years ago

After initially running the docker container and running a ghcc scan, the data from subsequent scans is not updated in the UI. I've experienced this problem numerous times. The first scan always works fine, but subsequent scans often does not update in the UI. Reviewing the Crawler Logs confirms the subsequent scans actually ran with new findings, but nothing is updated in the UI (not even the date of the last scan). I've even waited weeks between scans to ensure new commits occurred, but still the UI does not update.

jfalken commented 8 years ago

I was unable to repro this issue. For me, if new hits are found, they show in the UI.

A couple things to note here; if you need to restart the crawler to find more potential hits, click the refresh icon on the Process Supervisor page. This will restart the crawler and it will start where it left off. If the crawler finds new results, it will insert the record(s) into MongoDB. The Crawler Logs page will report an entry like:

[2016-01-15 17:08:51] [INFO] - [Github_Audit] Inserted 20 results into mdb. oid: 569927a30d560300b1ff61ca

At that point, when you refresh the results / Home page, you will see the new results.

Please note that the docker container was intended as a demo/trial. For continuous usage, I highly recommend you run the python script cron'd on a server, and determine how best to view the results (you could use the builtin web app if desired). The docker container app does not have a means to cron or restart the processes periodically.