search

mainnebula / SPACE_TASKS

1 stars 3 forks source link

Simple Python scraping script and SQL storage #1

Open mainnebula opened 5 years ago

mainnebula commented 5 years ago

Description I want a python script written that takes a database of satellite objects from this Celestrak Catalog and scrapes a list of the web-sources for description data of objects that match that end-point’s NORAD Catalog Number or International Designator.

The script will populate, and upon subsequent calls will update a SQL database table of URLs, keyed by NORAD Catalog Number, corresponding to valid links for additional object information.

Requirements

Other Info

The URL validation should target these sources in order and populate based on availability:

  1. NSSDC Example: https://nssdc.gsfc.nasa.gov/nmc/spacecraft/display.action?id=2019-025A

  2. Celestrak.com Example: https://celestrak.com/satcat/1968/1968-110.php#A

  3. Wikipedia Example: https://en.wikipedia.org/wiki/Explorer_1

  4. Gunters Satellite Page (ordered scraping may not be possible) Example: https://space.skyrocket.de/doc_sdat/westpac.htm

The intent is to build a list of “verified links” where the user can be referred to an external URL with more information about a specific catalog object.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 1.5 ETH (250.49 USD @ $166.99/ETH) attached to it.

iamonuwa commented 5 years ago

@mainnebula work is currently in progress here. CC: @gitcoinbot

mainnebula commented 5 years ago

Hi @iamonuwa,

Any updates on this issue?

Thanks! :)

iamonuwa commented 5 years ago

Hey, yes. I'll send in a PR for review today

imwatsi commented 5 years ago

I wrote a script for this issue, @mainnebula. links/links.py

frankchen07 commented 5 years ago

@mainnebula - frank from Gitcoin here :) @imwatsi has submitted a PR for this issue - have you taken a look?

frankchen07 commented 5 years ago

@iamonuwa - did you get your PR up? If not, it's been almost 2 weeks, and we'll have to remove you from the bounty. Please let us know!

mainnebula commented 5 years ago

@frankchen07 extended deadline and coordinating with @imwatsi. Thanks!

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 2 years, 4 months ago. Please review their action plans below:

1) imwatsi has started work.

Work submitted and reviewed. Now working on corrections.

Learn more on the Gitcoin Issue Details page.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 1.5 ETH (265.92 USD @ $177.28/ETH) has been submitted by:

  1. @imwatsi

@mainnebula please take a look at the submitted work:

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of 1.5 ETH (242.23 USD @ $161.49/ETH) attached to this issue has been approved & issued to @imwatsi.