search

coala / gci-leaders

A website showing Google Code-in information 🏆
https://gci-leaders.netlify.com
39 stars 45 forks source link

Fix bugs in scrape.js and pre2017.js #161

Closed li-boxuan closed 6 years ago

li-boxuan commented 6 years ago

scrape.js: Check if property repositories exists Fixes https://github.com/coala/gci-leaders/issues/160 pre2017.js: Fix wikipedia scraping Fixes https://github.com/coala/gci-leaders/issues/162 scrape.js: Catch exception for GitHub GraphQL Fixes https://github.com/coala/gci-leaders/issues/163

TravisBuddy commented 6 years ago

Travis tests have failed

Hey @li-boxuan, Please read the following log in order to understand the failure reason. It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

npm run build ``` > gci-leaders@1.0.0 build /home/travis/build/coala/gci-leaders > npm run bundle && npm run gather && npm run scrape && npm run planet && npm run main && npm run generate > gci-leaders@1.0.0 bundle /home/travis/build/coala/gci-leaders > NODE_ENV=production webpack --progress clean-webpack-plugin: /home/travis/build/coala/gci-leaders/out has been removed. 0% compiling 10% building modules 0/1 modules 1 active ...coala/gci-leaders/src/js/bootstrap.js                                          10% building modules 1/1 modules 0 active 10% building modules 1/2 modules 1 active ...ld/coala/gci-leaders/src/js/locale.js 10% building modules 1/3 modules 2 active ...d/coala/gci-leaders/src/js/twitter.js 10% building modules 2/3 modules 1 active ...d/coala/gci-leaders/src/js/twitter.js                                          10% building modules 3/3 modules 0 active 10% building modules 3/4 modules 1 active ...odules/jquery.i18n/src/jquery.i18n.js 10% building modules 3/5 modules 2 active ....i18n/src/jquery.i18n.messagestore.js 10% building modules 3/6 modules 3 active ...ery.i18n/src/jquery.i18n.fallbacks.js 10% building modules 3/7 modules 4 active ...uery.i18n/src/jquery.i18n.language.js 10% building modules 3/8 modules 5 active ...jquery.i18n/src/jquery.i18n.parser.js 10% building modules 3/9 modules 6 active ...query.i18n/src/jquery.i18n.emitter.js 10% building modules 3/10 modules 7 active ...build/coala/gci-leaders/src/js/app.js 10% building modules 3/11 modules 8 active ...ild/coala/gci-leaders/src/js/utils.js 10% building modules 4/11 modules 7 active ...ild/coala/gci-leaders/src/js/utils.js 10% building modules 5/11 modules 6 active ...ild/coala/gci-leaders/src/js/utils.js 10% building modules 6/11 modules 5 active ...ild/coala/gci-leaders/src/js/utils.js 10% building modules 7/11 modules 4 active ...ild/coala/gci-leaders/src/js/utils.js 10% building modules 8/11 modules 3 active ...ild/coala/gci-leaders/src/js/utils.js 11% building modules 9/11 modules 2 active ...ild/coala/gci-leaders/src/js/utils.js 11% building modules 10/11 modules 1 active ...ild/coala/gci-leaders/src/js/utils.js                                          11% building modules 11/11 modules 0 active 11% building modules 11/12 modules 1 active ...coala/gci-leaders/src/styles/main.css                                          11% building modules 12/12 modules 0 active 11% building modules 12/13 modules 1 active ...coala/gci-leaders/src/styles/main.css 11% building modules 12/14 modules 2 active ...modules/style-loader/lib/addStyles.js 11% building modules 12/15 modules 3 active ...uleParser/src/CLDRPluralRuleParser.js 11% building modules 13/15 modules 2 active ...modules/style-loader/lib/addStyles.js 11% building modules 13/16 modules 3 active ...rs/node_modules/jquery/dist/jquery.js 11% building modules 14/16 modules 2 active ...modules/style-loader/lib/addStyles.js 11% building modules 14/17 modules 3 active ...rs/node_modules/jquery/dist/jquery.js 11% building modules 15/17 modules 2 active ...modules/style-loader/lib/addStyles.js 11% building modules 15/18 modules 3 active ...rs/node_modules/jquery/dist/jquery.js 11% building modules 16/18 modules 2 active ...rs/node_modules/jquery/dist/jquery.js 11% building modules 16/19 modules 3 active ...uleParser/src/CLDRPluralRuleParser.js 12% building modules 17/19 modules 2 active ...uleParser/src/CLDRPluralRuleParser.js 12% building modules 18/19 modules 1 active ...uleParser/src/CLDRPluralRuleParser.js 12% building modules 18/20 modules 2 active ...de_modules/css-loader/lib/css-base.js 12% building modules 19/20 modules 1 active ...de_modules/css-loader/lib/css-base.js 12% building modules 19/21 modules 2 active ...ode_modules/webpack/buildin/global.js 12% building modules 20/21 modules 1 active ...ode_modules/webpack/buildin/global.js 12% building modules 20/22 modules 2 active ...node_modules/style-loader/lib/urls.js 12% building modules 21/22 modules 1 active ...node_modules/style-loader/lib/urls.js                                          12% building modules 22/22 modules 0 active                                 71% sealing 72% optimizing 73% basic module optimization       74% module optimization 75% advanced module optimization     76% basic chunk optimization       77% chunk optimization 78% advanced chunk optimization 12% building modules 22/23 modules 1 active ...coala/gci-leaders/src/styles/main.css                                          12% building modules 23/23 modules 0 active      79% module and chunk tree optimization         80% chunk modules optimization 81% advanced chunk modules optimization                     82% module reviving 83% module order optimization    84% module id optimization         85% chunk reviving 86% chunk order optimization    87% chunk id optimization               88% hashing 89% module assets processing  90% chunk assets processing 91% additional chunk assets processing                          92% recording 91% additional asset processing    92% chunk asset optimization       94% asset optimization           95% emitting             Hash: 96f75879fae921d9b6bb Version: webpack 3.10.0 Time: 5567ms Asset Size Chunks Chunk Names images/logos/global.png 1.75 kB [emitted] main.96f75879fae921d9b6bb.js 110 kB 0 [emitted] main main.96f75879fae921d9b6bb.js.map 531 kB 0 [emitted] main main.96f75879fae921d9b6bb.css.map 106 bytes 0 [emitted] main favicon.ico 15.1 kB [emitted] _headers 42 bytes [emitted] i18n/en.json 1.29 kB [emitted] i18n/es.json 1.16 kB [emitted] i18n/nb_NO.json 1.13 kB [emitted] i18n/pl.json 1.23 kB [emitted] images/chat.png 248 bytes [emitted] images/logos/gci.png 9.58 kB [emitted] images/logos/gitlab.png 5.77 kB [emitted] images/logos/gcilogo.png 21.4 kB [emitted] images/logos/gitter.png 1.66 kB [emitted] main.96f75879fae921d9b6bb.css 1.37 kB 0 [emitted] main images/logos/gplus.png 1.28 kB [emitted] images/logos/irc.png 2.51 kB [emitted] images/logos/LICENSE 78 bytes [emitted] images/logos/README.md 161 bytes [emitted] images/logos/rocket.png 3.21 kB [emitted] images/logos/slack.png 2.89 kB [emitted] images/logos/telegram.png 2.92 kB [emitted] images/logos/twitter.png 1.03 kB [emitted] images/logos/wikidata.png 3.37 kB [emitted] images/logos/wikipedia.png 11.8 kB [emitted] images/logos/zulip.png 2.54 kB [emitted] images/mail.png 420 bytes [emitted] images/rss.png 5.78 kB [emitted] manifest.json 202 bytes [emitted] [0] (webpack)/buildin/global.js 509 bytes {0} [built] [1] ./src/js/utils.js 2.16 kB {0} [built] [2] ./src/js/bootstrap.js 479 bytes {0} [built] [14] ./src/js/locale.js 1.21 kB {0} [built] [15] ./src/js/app.js 1.48 kB {0} [built] [16] ./src/js/twitter.js 2.58 kB {0} [built] [17] ./src/styles/main.css 41 bytes {0} [built] [18] ./node_modules/css-loader?{"minimize":true}!./node_modules/postcss-loader/lib?{"ident":"postcss"}!./src/styles/main.css 1.48 kB [built] + 14 hidden modules Child extract-text-webpack-plugin node_modules/extract-text-webpack-plugin/dist node_modules/css-loader/index.js??ref--1-2!node_modules/postcss-loader/lib/index.js??postcss!src/styles/main.css: [0] ./node_modules/css-loader?{"minimize":true}!./node_modules/postcss-loader/lib?{"ident":"postcss","plugins":[null]}!./src/styles/main.css 1.48 kB {0} [built] + 1 hidden module > gci-leaders@1.0.0 gather /home/travis/build/coala/gci-leaders > node ./lib/gather.js Could not gather data.min.json > gci-leaders@1.0.0 scrape /home/travis/build/coala/gci-leaders > node ./lib/scrape.js No existing data... Could not find GitHub org for Apertium. Resorting to GitHub API hit. Could not find GitHub org for Catrobat. Resorting to GitHub API hit. Could not find GitHub org for CCExtractor Development. Resorting to GitHub API hit. Could not find GitHub org for Copyleft Games. Resorting to GitHub API hit. Could not find GitHub org for Digital Impact Alliance. Resorting to GitHub API hit. Could not find GitHub org for Drupal. Resorting to GitHub API hit. Could not find GitHub org for Fedora Project. Resorting to GitHub API hit. Could not find GitHub org for Haiku. Resorting to GitHub API hit. Could not find GitHub org for KDE Community. Resorting to GitHub API hit. Could not find GitHub org for Liquid Galaxy project. Resorting to GitHub API hit. Could not find GitHub org for MetaBrainz Foundation. Resorting to GitHub API hit. Could not find GitHub org for OpenMRS. Resorting to GitHub API hit. Could not find GitHub org for OpenWISP. Resorting to GitHub API hit. Could not find GitHub org for OSGeo. Resorting to GitHub API hit. Could not find GitHub org for PostgreSQL. Resorting to GitHub API hit. Could not find GitHub org for Public Lab. Resorting to GitHub API hit. Could not find GitHub org for RTEMS Project. Resorting to GitHub API hit. Could not find GitHub org for Sugar Labs. Resorting to GitHub API hit. Could not find GitHub org for The ns-3 Network Simulator Project. Resorting to GitHub API hit. Could not find GitHub org for Wikimedia. Resorting to GitHub API hit. > gci-leaders@1.0.0 planet /home/travis/build/coala/gci-leaders > node ./lib/planet.js module.js:540 throw err; ^ Error: Cannot find module '../out/data.json' at Function.Module._resolveFilename (module.js:538:15) at Function.Module._load (module.js:468:25) at Module.require (module.js:587:17) at require (internal/module.js:11:18) at Object. (/home/travis/build/coala/gci-leaders/lib/planet.js:1:76) at Module._compile (module.js:643:30) at Object.Module._extensions..js (module.js:654:10) at Module.load (module.js:556:32) at tryModuleLoad (module.js:499:12) at Function.Module._load (module.js:491:3) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! gci-leaders@1.0.0 planet: `node ./lib/planet.js` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the gci-leaders@1.0.0 planet script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! /home/travis/.npm/_logs/2018-10-26T06_08_48_793Z-debug.log npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! gci-leaders@1.0.0 build: `npm run bundle && npm run gather && npm run scrape && npm run planet && npm run main && npm run generate` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the gci-leaders@1.0.0 build script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above. npm ERR! A complete log of this run can be found in: npm ERR! /home/travis/.npm/_logs/2018-10-26T06_08_48_803Z-debug.log ```
TravisBuddy Request Identifier: 9b0f4620-d8e5-11e8-b9f2-b1fbc111596d
li-boxuan commented 6 years ago

Sometimes I encounter the issue (node:1599) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): TypeError: Cannot read property 'line' of undefined. See https://app.netlify.com/sites/gci-leaders/deploys/5bd2b2ddb13fb157f7a0a066

Probably an upstream bug with graphql-client. Retry sometimes solves the problem.

UPDATE:

So from my experiment and observation, it is a bug with upstream project graphql-client.

https://github.com/nordsimon/graphql-client/blob/f8d1dcba8a65f290e35b71c883a18094594d6d5a/index.js#L2

can lead to an error when elements in errors array don't have locations field.

However, GitHub is supposed to include the locations field when returning errors.

For example, a normal errors should be sth like this:

[{"message":"Could not resolve to an Organization with the login of 'GCI-AOSSIE'.","type":"NOT_FOUND","path":["organization"],"locations":[{"line":2,"column":3}]}]

However, sometimes GitHub returns this:

[{"message":"Something went wrong while executing your query. This may be the result of a timeout, or it could be a GitHub bug. Please include `E689:1AF6:67E816:BAAF4C:5BD321E1` when reporting this issue."}]

which causes the error of graphql-client.

CONCLUSION: a bug of upstream project graphql-client. It only happens occasionally, coz it's caused due to some unknown error of GitHub.

UPDATE 2:

Created an issue https://github.com/coala/gci-leaders/issues/163

li-boxuan commented 6 years ago

@andrewda Yep, this is to be merged after #157.

jayvdb commented 6 years ago

@li-boxuan , can you add a commit to fix the year. We gotta get this up . Can't wait any longer.

li-boxuan commented 6 years ago

@jayvdb Done. Without moban it is difficult to make the year in static files dynamic.

utkarsh2102 commented 6 years ago

Hey, @li-boxuan, I've made it dynamic. Please check: https://github.com/coala/gci-leaders/pull/157 CC: @andrewda @jayvdb

li-boxuan commented 6 years ago

@utkarsh2102 Thanks. I've rebased this PR on top of yours.

jayvdb commented 6 years ago

ack https://github.com/coala/gci-leaders/pull/161/commits/e204e472f8d2ca15956d606bd937adad6750d044 https://github.com/coala/gci-leaders/pull/161/commits/ae47a1835b87988a81a21ab9ae682c219f93a7aa https://github.com/coala/gci-leaders/pull/161/commits/2ea74bc25661ac8f20028555df1d268a780f0c9f

jayvdb commented 6 years ago

@gitmate-bot ff