openzim / mindtouch

libretexts.org to ZIM scraper
GNU General Public License v3.0
0 stars 0 forks source link

Add API and caching #4

Closed benoit74 closed 2 weeks ago

benoit74 commented 3 weeks ago

Changes:

Nota: this is not yet used to populate anything in the ZIM, only "utility" code for now

To be reviewed after https://github.com/openzim/libretexts/pull/3 is merged

codecov[bot] commented 3 weeks ago

Codecov Report

Attention: Patch coverage is 17.85714% with 92 lines in your changes missing coverage. Please review.

Project coverage is 44.94%. Comparing base (8efbbe5) to head (40cb342). Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
scraper/src/libretexts2zim/client.py 19.76% 69 Missing :warning:
scraper/src/libretexts2zim/entrypoint.py 16.66% 10 Missing :warning:
scraper/src/libretexts2zim/processor.py 11.11% 8 Missing :warning:
scraper/src/libretexts2zim/__main__.py 0.00% 5 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #4 +/- ## =========================================== - Coverage 56.72% 44.94% -11.79% =========================================== Files 7 7 Lines 238 336 +98 Branches 19 32 +13 =========================================== + Hits 135 151 +16 - Misses 102 184 +82 Partials 1 1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

benoit74 commented 2 weeks ago

I find we're duplicating the cache reading/writing bits ; maybe room for sharing this

This is probably because some of the code was buggy ^^ I fixed this. There is indeed only small variations ... but pretty important ones.

benoit74 commented 2 weeks ago

I had to rebase to fix conflicts. Please review only two last commits which contains "real" modifications.

Nota: Iast commit is about making codecov informational. I find it's mostly mandatory since in scrapers like this one, most code is covered by integration tests, but these are not reported in codecov ...