Dependencies on dweb.me or dweb.archive.org

mitra42 commented 5 years ago

It would be better if dweb-mirror wasn't dependent on dweb.me, that could involve a) calling archive.org directly where appropriate, b) a + performing data manipulation on return rather than in dweb.me c) microservices running on dweb.me that dont depend on the core deb.me server

Next comment will be list of places

(Scroll down to "ROUND 4") but all issues now have own topics so closing

mitra42 commented 5 years ago

Metadata - see comment below DONE

[x] torrent file editing
[x] CORS ? though dweb-mirror not subject to it - handled by dweb-metadata
[x] Search dweb-mirror code for dweb.me and dweb.archive.org
- [x] "dweb.me" - only appears in config.yaml/upstream
- [x] Search on "upstream" < proxyUpstream
  - /arc/archive.org/metadata/IDENTIFIER/FILENAME look at how python edits this see ArchiveFile comment below
  - /contenthash/* - see below
[x] check IAUX for "dweb.me" "dweb.archive.org" "upstream"
[x] check dweb-archive for "dweb.me" "dweb.archive.org" shouldnt be any as should use gateway
- [x] but may need to check that dont depend on any changes,
- [x] and that URLs used are not dweb.me or dweb.archive.org specific
[x] check dweb-archivecontroller for "dweb.me" and "upstream"
Replace things Python does ... see below

Note this is in parallel to any work done in dweb-transports to split it up and re-modularize it.

mitra42 commented 5 years ago

Archive.py / ArchiveItem DONE

ArchiveItem._enforceMetadataContracts (see metadata below)
ArchiveItem.new
- See item2thumbnail() - see comment below
- [x] Think about .____padding_file/123 download handles this fine
- [x] check cors issues
ArchiveItem.metadata - see metadata below
ArchiveItem.leaf - not used anywhere
ArchiveItem.item2thumbnail() - see comment below.
ArchiveItem.thumbnail - doesnt appear to be used anymore, use /services/img

mitra42 commented 5 years ago

Archive.py ArchiveFile < /arc/archive.org/download (which goes via ArchiveItem I think)

ArchiveFile.new
- [x] adds a multihash field, can make this on client if needed (only for ContentHash: which not really using
- [x] magnet links, if used can be built from items magnet link
AF.cache_content - used by DA but not DM
[x] httpUrl() goes to gatewayServer - see comments on it

mitra42 commented 5 years ago

Archive.js / AdvancedSearch < /arc/archive.org/advancedsearch

AdvancedSearch.new
- sets thumbnaillinks - not really reqd for DM, see item2thumbnail() below
- [x] sets collection0title - see collectiontitle()
- [x] lobby Aaron to include it,
- sets collection0thumbnaillinks - see item2thumbnail below
- AdvancedSearch.collectiontitle() has heuristic and call
- [x] build into Fjords as call and cache

mitra42 commented 5 years ago

[x] Replace things that python does
- Archive.py
  - [x] sortorder & parentsortorder & AI._collectionsortorder() - build into Fjords - note might end up in metadata at some point :-)
  - [x] ArchiveSearch - see other comment
  - [x] ArchiveFile - see other comment
  - [x] ArchiveItem - see next comment
- Btih.py - see below
- HashResolvers
- Sha1Hex < url sha1hex < ServerGateway < NOT USED
- ContentHash , see comment below
- HashFileEmpty (special case of empty hash)
- ServerGateway
- "advancedsearch": AdvancedSearch see above (other comment)
- [x] "archiveid": ArchiveItem,
- "doi": DOI, OBS
- "contenthash": ContentHash, see comment below
- "sha1hex": Sha1Hex, Not used
- "rawstore": LocalResolverStore, only used by examples
- "rawfetch": LocalResolverFetch, only used by examples (local files, not IA)
- "rawlist": LocalResolverList, only used by examples
- "rawadd": LocalResolverAdd, only used by examples
- "btih": BtihResolver, see comment below
- "table": KeyValueTable, only used by examples
- /arc/archive.org/
  - download |server- see ArchiveFile comment (goes via ArchiveItem)
  - advancedsearch see advancedsearch comment
  - leaf NOT USED (I dont think)
  - services/img || thumbnail: -> AI.thumbnail() see item2thumbnail below
  - torrent -> AI.torrent() (see ArchvieItem above)
  - metadata -> AI.metadata (see ArchiveItem above)
- Scan code (from python/ServerGateway.py) see commen below

mitra42 commented 5 years ago

ContentHash

< url contenthash < Done locally on mirror that then asks dweb.me,
believe direct search is blocked from non-archive hosts, but
mimimal value to this.< DAC uses it, which means DM will check for a local copy
[LATER] Could have DM check for hash in metadata, then look up contenthash, then get locally/elsewhere, but super low priority
[x] Still in DAC but no value

mitra42 commented 5 years ago

/arc/archive.org/metadata and ArchiveItem.metadata

[ D.A.O] Add IPFS addresses, not needed by dweb-mirror
[x] Only de-fjording is in Archive.js/_enforceMetadataContracts to single/array - not reqd as done in D.A.C.
[x] magnet links getting edited by dweb.me - See setmagnetlink() - moved to www-dweb-metadata service
[x] see item2thumbnail() - since not using WT (not effective for small file in large torrent) for thumbnail and not using IPFS (reliabiity) then this doesnt add much to do on server side.
- [x] remove dependency on DA/DM side
  - [x] TEST works for clicking into local, settings (it doesnt because no image displayed there anyway)
[x] _enforceMetadataContracts duplicates rule set in DAC
replace assumption about collection_titles with lookup from DA->DM which can cache it - see #283
[x] TEST replaced assumptions about collection_sort_order with heuristics - in AI and MH call

mitra42 commented 5 years ago

[x] Archive.py ArchiveItem.setmagnetlink - replaced by dweb-metadata service
[x] Archive.py ArchiveItem.modifiedtorrent - replacd by dweb-torrent service
[x] dweb.me/...torrent... -> AI.torrent() (see ArchvieItem above) obsoleted consumers go to dweb-torrent service
AF.inTorrent
- [x] Notice check for date inconsistency (file newer than torrent) make sure DA/DM knows not to use in that case

mitra42 commented 5 years ago

[x] Archive.py / ArchiveItem: item2thumbnail()
this stores in IPFS, and returns IPFS address as part of metadata, given unreliability of IPFS (things we've stored disappear, aren't in the global DHT, so are single point of failure) this might not be worth it, since adds significant latency on especially first fetch.
new dweb-metadata service does NOT currently do this IPFS store - low priority to fix.
DM doesn't need it for DA could build a service if want.
DA no longer looks at thumbnaillinks, so not relevant any more

thumbnaillinks

Added in search, to Members, but search expands -
[x] On Archive.py: Search sets collection0thumbnaillinks - not used any more
[x] Check DA, DM, IAUX, DAC for any references
- [x] AI.metadata.thumbnaillinks < AI.thumbnaillinks() < RS.thumbnailUrlsFrom < UNUSED REMOVE
- [x] AM.metadata.thumbnaillinks
  - [x] < AM.urls() ALSO GOES TO services/img SIMPLIFY AM.urls()
  - [x] < AM.p_urls REMOVE < p_resolveUrls()
  - [x] < AMP.saveThumbnail() USE AM.urls
  - [x] < CrawlItem.process x2 (does fetch_metadata for Tile case, not needed if know how to get thumbnail elsewhere in both AMP.saveThumbnail and AI.saveThumbnail) FIX
  - [x] AI.thumbnaillinks() << OBS
- [x] Util.specialidentifiers.*.thumbnaillinks < AMP.streamThumbnail Its different field ...
- [x] DA & DM DAC lot of docs for thumbnaillinks
- [x] AM.metadata.collection0thumbnaillinks < IAUX.TileComponent Go to /services/img

[x] ServerGateway/arc/archive.org/.../thumbnail services/img || thumbnail: -> AI.thumbnail()
- check for any calls to that URL - NOT USED

mitra42 commented 4 years ago

Python/ServerGateway

/info as heartbeat - see Cleanup
A scan of code suggests everything else handled

mitra42 commented 4 years ago

Code rescan

Done: DAC; IAUX DA, DM, dweb-torrent, dweb-metadata for dweb.me dweb.archive.org upstreamPrefix gatewayServer httpUrl

Strategy

[x] Enable a "mirror" version of the domain config
[x] create a way in config to enable it (already done via "mirror")
[x] make sure archive.html as mirror enables it
[x] create a combo DTS.httpFetchUrl(DTS.resolveNames(...)) for cases not going to come back through DWT (this is AF.httpUrl
[x] go through places below and replace as/when reqd
[x] Look for TODO-DM242 in DAC
[x] Look for cases of DwebArchive.mirror esp in DAC
[x] Rescan dweb.me dweb.archive.org mds/ upstreamPrefix, gatewayServer httpUrl esp for use in docs

Places

[x] This assumes translator can divert if mirror set - that needs adding to Naming.js
[x] DTS.p_resolveNames -> DTS.gatewayUrls | DTS.gatewayUrl
[x] DAC.upstreamPrefix which is exported so check other repos
- [x] DAC.AF.httpUrl
- [x] DAC.AF.urls
  - [x] http url - standard
  - magnet - see below
  - metadata/IDENTIFIER/NAME for IPFS link - see below
- [x] DA.AudioVideo.render for url for poster
- [x] DA.RadioPlayerController for new AudioSource which presumably doesnt go thru DTS
- [x] DA.RadioPlayerController for waveFormImageUrl
- [x] DAC.AI._fetch_metadata - move to https://archive.org as then translated in Naming.js
- [x] DA.ReactSupport.preprocessDescription -go to root relative urls
[x] DAC.gatewayServer which is exported so check other repos
- [x] DAC.AF.urls which goes to dweb.archive.org to get IPFS urls
- [x] DAC.AI._fetch_metadata to fetch playlist - move to https://archive.org and then check translated in Naming.js
- [x] DAC.AI._fetch_bookread
- [x] bookreader/BookReader
- [x] and to specify"server" for JSIA call (think through)
- [x] DAC.AI.relatedItems - to go through be - move to https://archive.org and then check Naming.js
- [x] DAC.AM.httpUrl for /services.url should always be able to go to archive.org and get translated
- [x] DA.archive.html passed as http.urlbase to dweb-transports - think
- DA.reactSupport.transportStatusAndProps for /info URL - see below
- [x] DA.BookReaderDwebWrapper
- [x] to get server to pass to BookReaderWrapper.jsia - think
- [x] to pass to imagesBaseURL - think
- [x] DA.DetailsIAWrap x2 as props on movies theatre - think
- [x] DA.RadioPlayerController for logoUrl - need to map here as prob not going through dweb-transports
- [x] DA.SaveModal.SaveDirectory for /admin/crawl/add will always be mirror, but going through DTS so could use Naming
- [x] DA.LocalComponent.LocalItem
- [x] for /info to get config from mirror - needs to be mirror
- [x] for byline
- [x] DA.SettingsComponent - 3 places for urls it uses to mirror
- [x] DAC.AM.urls
- [x] DAC.Util _query move to archive.org and check translated
- [x] DM.AIP.saveThumbnail - could go to archive.org as thru DTS
[x] IAUX.util.gatewayServer
- [x] IAUX.CrawlConfig to go to admin/setconfig which will always be on the mirror
- [x] IAUX.getLanguage probably can go to Naming but needs translating for dweb.archive.org or other static server
[x] IAUX.util.canonicalUrl
- [x] IAUX.DetailsCollectionList (services/image) so could go a.o/services/image and be translated
[x] IAUX.AnchorDownload.constructor should be able to go to a.o/download or cors.a.o/download
DA.Page.jsx.AJS.date_switcher x3 see below
DM.configDefaults.yaml.connect.webtorrent.trackers - see above
[x] DM.configDefaults.yaml.upstream
- [x] mirrorHttp.proxyUpstream - see what used for, and if can use DT.resolveNames etc

mitra42 commented 4 years ago

Torrent file handling

[x] DAC.AF.urls
- [x] Make dweb-torrent support /download/IDENTIFIER/IDENTIFIER_archive.torrent
- [x] Use that url in AF.urls
- TOO HARD: Make naming/mirror catch it
- requires regex
- [x] No need for it in naming, as already point at microservice

mitra42 commented 4 years ago

metadata/IDENTIFIER/NAME

[x] DAC.AF.urls probably wont work - as will end up at dweb-metadata which will fail.
- [x] Maybe some special cases in DTS.Naming.js to catch this and send to dweb.archive.org for now
[x] mirrorHttp goes to proxyUrl explicitly till this is fixed in dweb-metadata or elsewhere
[x] Code disabled see https://github.com/internetarchive/dweb-archivecontroller/issues/11

mitra42 commented 4 years ago

/info

[x] DTS.TransportHTTP defaults to dweb.me and checks it for /info - not harmful left as is for now
[x] some other places as well ...
[x] archive.html goes to mirror or dweb.archive.org - not harmful as now
[x] DM.ReactSupport.transportStatusAndProps - mirror or dweb.archive.org not harmful as now

Note DA.SettingsComponent and DA.LocalComponent go to mirror correctly

mitra42 commented 4 years ago

ROUND 3

[x] /info as heartbeat - see above but should be ok
Static files served by dweb.archive.org (inc archive.html, languages, includes etc) see below
- [x] all now on www-dweb.dev.archive.org
- [x] /archive/ forwawrds in DM see below
[x] Thumbnail in BookReaderWrapper.render - works in offline, not going decentralized because render is sync, could be rewritten using "loading" pattern, but not really worth it - flagged in code if want to tackle later.
[x] Date switcher see https://github.com/internetarchive/dweb-archive/issues/176
- Not a blocker - i.e. issue exists, but doesnt change with move to JS based dweb.archive.org
[x] waiting on collection0title from Aaron see #283
- Not a blocker, for now we'll do another metadata fetch after each search
[x] Make dweb-torrent and dweb-metadata more modular, so can be easily combined or separate
[x] Consolidate dweb2 to use microservices see comment below
[x] Rip out Python code .... see below

All these below moved to ROUND 4 below

[ ] Have dweb.archive.org add IPFS addresses in metadata/IDENTIFIER/NAME - not doing as too slow and maybe put in Gun see https://github.com/internetarchive/dweb-archivecontroller/issues/10
[ ] Tracker is dweb.archive.org:6969 (set in DAC.mungeTorrent) need to move to own repo or something also DM.configDefaults.yaml see above and see #292
[ ] Allow seeder & tracker not to be dependent on dweb.me see BTIH #292 - no longer dependent on dweb.me but deployed in dweb.me
[ ] Rescan for TODO-DM242 but should all be listed in this comment
[ ] consider moving /iiif to its own repo so could be used on a microservice - not really needed yet

mitra42 commented 4 years ago

FAILS

STR

[x] (http://localhost:4244/details/geometry_dash_1.9)
- click play - starts alright
- click 2nd track
- fails on both image and mp3 Uncaught (in promise) DOMException: Failed to load because no supported source was found.

Diagnosis

[x] https://www-dweb-cors.dev.archive.org/cors/geometry_dash_1.9/Geometry%20Dash%20OST/BlastProcessing.png "Expectation Failed"
[x] https://archive.org/cors/geometry_dash_1.9/Geometry%20Dash%20OST/BlastProcessing.png
- File not found: /17/items/geometry_dash_1.9/Geometry%20Dash%20OST/BlastProcessing.png
[x] Why did dweb-cors mangle error on return
[x] Is /cors really the right URL (YES) and where did it come from (ROUTING)

This appears to be fixed, not sure how but prob in fixing other routing issues.

mitra42 commented 4 years ago

Static Files

Files served by dweb.archive.org (inc archive.html, languages, includes etc) see below

currenty eg. dweb.archive.org/languages -> in nginx /archive/languages/...
[x] build dweb2 repo to do mostly static serving
https://www-dweb.www-dev-archive.html ->
https://www-dweb.dev.archive.org/details/home etc working

Cleanup

[ ] consider as Express service instead with other modules

mitra42 commented 4 years ago

remove /archive/ forwards from DM.mirrorHttp

[x] work directly with those files as on archive.org and www-dweb.dev.archive.org

mitra42 commented 4 years ago

Consolidate dweb2 to use all new microservices

[x] dweb-torrent: btih and /download/ID/ID_archive.torrent
[x] dweb-metadata: /metadata/ID
[x] dweb-cors: ALl the cors add in
[x] dweb2: Static files

mitra42 commented 4 years ago

Rip out python code

[x] What still uses it ?
- [x] Check DAC.routing (done - see other items in this list)
- [x] Check dweb-mirror (Only goes to wss on 6969 for ??? (not python) (same thing in DAC.mungeTorrent)
- [x] check DA, IAUX, DAC, dweb-torrent, dweb-metadata, dweb-cors, dweb-transport, dweb2, dweb - all good except as listed
- [x] DTS look at use of urlbase, defined in DA.archive.html._transportOptions, used for /info and for TransportHash (which isnt currently used anywhere.
- [x] Check TODO-DM242 (see ROUND3) -currently only in DAC, DA and none still relevant to this
- [x] GUN still uses it - but will be fine once dweb2 running as dweb-archive.org
- [x] DTS.TransportHTTP.updateInfo - used for info - dweb2 will need to respond on /info like deb.archive
[x] Test static site can run 100% off https://www-dweb.dev.archive.org
[x] Reconsolidate dweb2 into the dweb repo (see comment above)
[x] DAC.routing -> contenthash (maybe base on the btih lookup in dweb-torrent - obsoleted
[x] DAC.routing -> examples - bet probably none of them work anyway - obsoleted
[x] Wolk -> dweb.archive.org/metadata - should stay same when move dweb2 to dweb.archive.org
[x] DA.Page.js -> date_switcher see internetarchive/dweb-archive#176 (will move over)
[x] IAUX.util.js.canonicalUrl for /services, either a) move to a.o or b) obs code

Python code is now gone .... both www-dweb.dev.archive.org and dweb.archive.org run on Javascript express servers now.

mitra42 commented 4 years ago

dweb2 as dweb - debugging

[x] TEST https://www-dweb.dev.archive.org/details/commute gets cors error on 'https://www-dweb-metadata.dev.archive.org/metadata/commute/commute.mp4' from origin 'https://www-dweb.dev.archive.org'
- disable IPFS check for now - see elsewhere in this issue
[x] Its not bring http up since dweb not starting ... over to fix that ....

dweb repo

From kubectl logs -n$NS pod/$POD

[ ] Not starting ipfs - looks like ipfs-update is failing again, posted an issue, disabling IPFS for now. https://github.com/ipfs/ipfs-update/issues/114
[X] Not starting redis - looks like only half removed - commented out in supervisor as no longer using
[X] dweb-express (exit status 1; not expected) - problem with dependencies
[x] TEST https://dweb.archive.org/metadata/commute fails with DwebTransports is not defined
[x] TEST https://dweb.archive.org/serve/commute/setup.gif failing - same reason as metadata/commute
[x] https://www-dweb.dev.archive.org/metadata/commute - Cannot GET /metadata/commute
- [x] probably related on /serve/commute/setup.gif. and same on dweb.archive.org
- routed() needed to handle root-relative urls

mitra42 commented 4 years ago

ROUND 4 - conclusion

Status

www-dweb.dev.archive.org is running just javascript under express
dweb.archive.org runs same JS along with the other super-peers (Gun; Wolk; Webtorrent Seeder/Tracker; IPFS).

Current oustanding issues will get their own issues.

[ ] ipfs-update was failing see https://github.com/internetarchive/dweb-archivecontroller/issues/10
[ ] metadata/IDENTIFIER/FILENAME (to get IPFS hash) not yet supported, deprioritised because of ipfs-update and list of other reasons meaning it doesn't get used - see https://github.com/internetarchive/dweb-archivecontroller/issues/11
[ ]

internetarchive / dweb-mirror