[Geo data] How can I contribute developer's geographic data to open-digger

[PureNatural]

Description

I currently have about 1 million developer's geographic location information, and there will be more data to be parsed in the future.

How can I submit this data to open-digger? @frank-zsy

[frank-zsy]

@PureNatural Thanks, I am curious about what is developers' identification? GitHub users or other platforms? And what is the geographic location information format?

Currently, OpenDigger contains about 215 thousands users with location information on GitHub.

The location information in OpenDigger right now is:

• location: the original location user provided.
• country: the country of the location
• administrative_area_level_1: the province or state name of the location
• administrative_area_level_2: the city of the location
• locality: the detailed district or street of the location

We will try our best to parse the location into detailed info, so the administrative_area_level_1 and more detailed fields may be null.

[srsxyc]

Developers are GitHub users. The location information is:

• adminDistrict: the province or state name of the location
• adminDistrict2: district level
• countryRegion: the country of location
• formattedAddress: formatting of geolocation information
• latitude
• longitude The above are parsed from the location. Also to get as much geolocation information as possible, we also parsed the email addresses of GitHub users to get some country-level geolocation information. Currently there are roughly 660,000 processed data, and this raw data is stored in json files. If you need guidance on the specifics of the data, I can show you the json file first. @frank-zsy

[frank-zsy]

@srsxyc That will be really great if you give me the JSON file and I will import it into ClickHouse for further use. A few more questions to confirm:

• When did these data collected?
• How did you parse the user provided location info to Geo info? I used Google Geo location API to do the work.

[srsxyc]

The data source we used was the GitHub log data from 2015 to present that you provided. We'll start by getting the de-duplicated usernames from the log data. Then after crawling the user's information through the GitHub API. Finally it is parsed through Bing MAP API. @frank-zsy

[frank-zsy]

Great, can you give me the JSON file and I can import into ClickHouse. By when I mean users may change their location information any time, so I need to know when did you call the GitHub API and retrieve the data from GitHub.

And are latitude and longitude often used for analysis? I can add the columns too.

[srsxyc]

We've only crawled data from 3 million users so far, with the earliest being roughly March 2023 and the latest being roughly May 2023. Because there is still a large amount of data that has not been captured, we have not updated it.

Latitude and longitude is an important piece of information when performing geolocation analysis. I think it's best to keep it.

How do I give you the JSON file?

[frank-zsy]

@srsxyc Any form you like, send me by WeChat after compress, or Baidu pan, or upload it to OSS and share the link. All is fine with me.

[frank-zsy]

@srsxyc Thanks for the data, all the users data has been insert into the gh_user_info table and I will insert the location info to location_info table too.

[frank-zsy]

@srsxyc All the data right now have been updated into the ClickHouse, thanks a lot.