Need to record this somewhere... the earliest history of Podping

Podcastindex-org / podping.cloud

The server code that runs a podping.cloud node.

MIT License

36 stars 11 forks source link

Need to record this somewhere... the earliest history of Podping #35

Open brianoflondon opened 3 years ago

brianoflondon commented 3 years ago

@podping_stats_toot_bot @dave this is where Dave started testing with his websub feed:

https://hiveblocks.com/tx/f0affd194524a6e0171d65d29d5c501865f0bd72 -> 2021-05-07 20:58:33 (UTC)

I think this is somewhere near when RSS.com came on board

https://hiveblocks.com/tx/f8aa1880e913efe50c954675f8575f9f363a10d0 -> 2021-05-18 15:36:21 (UTC)

Which matches up with Dave's toot:

https://podcastindex.social/@dave/106256933320987110

seakintruth commented 3 years ago

I'm wondering what the best way to handle this is.. here are some approaches i'm considering.

In my call to hive-watcher.py ensure that I never search past this time, with some bash math.
Add a check to the hive-watcher.py's history search to stop prior to this timestamp or block, and add an argument --include-alpha (defaults to false) to allow for ignoring this check

seakintruth commented 3 years ago

I'm torn, but updating the hive-watcher is probably the wrong move, as it shouldn't be making this check for ever.

never look past the timestamp: 1621352180 as it's the first valid feed ping. https://hiveblocks.com/tx/f8aa1880e913efe50c954675f8575f9f363a10d0

Then again the juice is probably not worth the squeeze in either case ... so what if a small fraction of this month's posts aren't valid...

brianoflondon commented 3 years ago

There is a project which is working on holding all custom_json in a database.

Might be something you can work with him on.

https://peakd.com/hive-139531/@imwatsi/hive-plug-and-play-is-now-live

seakintruth commented 3 years ago

Thanks for this, I had just started building this out for podping records out as a sqlite3 local database file, this project puts everything in a more robust postgresql dabase

I'm considering forking it and running my own node dedicated to podping.cloud, pre-filtered for only podping, and adding additional podping specific tables. This would reduce total on disk requirements two orders of magnitude. And setting the option to this block: start_block=54000984

https://github.com/imwatsi/hive-plug-play/wiki/Run-your-own-node

This could provide a micro-service as an API with end points that may be useful to podcatchers that would like to get caught up quickly. (currently this project is claiming to be running 10 blocks behind the block chain)

seakintruth commented 3 years ago

I've put this info on the readme for the postgresql database rest project fork for podping