gridcoin-community / Gridcoin-Research

Gridcoin-Research
MIT License
586 stars 172 forks source link

Neural Network Can't Download Stats from World Community Grid #323

Closed ghost closed 7 years ago

ghost commented 7 years ago

WCG was not in the NN this morning.

Since WCG moved to the IBM cloud it looks like they have changed the URL's to their stats from user.gz to user.xml.gz and team.gz to team.xml.gz.

It looks like the extensions are hardcoded into the VB.NET

The logs read:

May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Team world community grid http://www.worldcommunitygrid.org/boinc/stats/team.gz
May 23 2017 00:11, Error while downloading master team gz file: The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
May 23 2017 00:11, Error while downloading master project rac gz file : The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
May 23 2017 00:11, Error while downloading master project rac gz file : The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
May 23 2017 00:11, Error while downloading master project rac gz file : The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
May 23 2017 00:11, Error while downloading master project rac gz file : The remote server returned an error: (404) Not Found., Retrying.
May 23 2017 00:11, Syncing Project world community grid http://www.worldcommunitygrid.org/boinc/stats/user.gz
ghost commented 7 years ago

@tomasbrod it looks like the reason for the file names not being in the beacon is because there are two files to download.

  1. The team.xml.gz
  2. The user.xml.gz

The team.xml.gz is downloaded to find the Gridcoin team ID.

The team ID is then used to calculate magnitudes for users with that team ID.

I assume the reason for not specifying both URL's in the beacon was to save space. A possible alternative would have been to store the team ID in the beacon so that the team.xml.gz file does not need to be downloaded at all. Then the full path to the user.xml.gz file could be used.

e.g

<WHITELIST>Amicable_Numbers<COL>http://sech.me/boinc/Amicable/stats.user.xml.gz<COL>[TEAMID]</WHITELIST>`

// INSTEAD OF
<WHITELIST>Amicable_Numbers<COL>http://sech.me/boinc/Amicable/@</WHITELIST>

With the team requirement being removed the problem goes away providing the project beacons are updated with the full paths to user.xml.

moisespr123 commented 7 years ago

Sounds like a great idea. Definitely only downloading the users.xml.gz file would help save precious bandwidth for both us and the BOINC project server. I believe to do this, there would need to be a mandatory release right? Making the old NN obsolete since it would have an additional field that the old NN would not understand.

moisespr123 commented 7 years ago

So, is this fixed in the .9.0 release?

denravonska commented 7 years ago

@moisesmcardona Yup, for now :)