search

Hamuko / cum

comic updater, mangafied
Apache License 2.0
170 stars 15 forks source link

dynastyscans: Scrape using JSON endpoints #40

Closed CounterPillow closed 7 years ago

CounterPillow commented 7 years ago

dynasty-scans.com has largely undocumented JSON endpoints, which this change makes use of. Instead of parsing through fragile HTML, we can now enumerate the data we need from the JSON endpoints.

This reduces the amount of requests we need to make, particularly for the get_groups method. It also reduces code complexity somewhat, as we now no longer need to iterate through the HTML DOM multiple times to get all the info we need.

For now, headings such as "Volume 1" are skipped. These could be used for volume specific shenanigans in the future if need be.

codecov-io commented 7 years ago

Codecov Report

Merging #40 into master will not impact coverage by -0.13%.

@@            Coverage Diff             @@
##           master      #40      +/-   ##
==========================================
- Coverage   97.64%   97.52%   -0.13%     
==========================================
  Files          16       16              
  Lines        1487     1492       +5     
==========================================
+ Hits         1452     1455       +3     
- Misses         35       37       +2
Impacted Files Coverage Δ
cum/scrapers/dynastyscans.py 96.59% <93.47%> (-1%) :x:
cum/scrapers/batoto.py 94.57% <ø> (-0.78%) :x:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 207d29e...9409342. Read the comment docs.