Fixing error on building the project, updating the README for ease of installation

[korymath]

Hi, thank you for the great work on the library. I love this sort of visualization and have used your tool at least once a week for the last year or so. I wanted to support the development by attempting to build on my person OSX system. I ran into some snags, and so I wanted to list them here to make it easier for the next person and to help you by continuing to support the dev.

I would update the READ to ensure that the preliminary requirements are covered.

For instance, that yarn is installed:

brew install yarn

Error run code: cargo run --release ... similar to this issue: https://github.com/mimblewimble/grin/issues/2339 ... The solution was found by digging through here: https://github.com/rust-lang-nursery/lazy-static.rs/issues/114#issuecomment-413266021

And it was related to a version issue, so make sure that the versions are updated and accessible from the given CLI.

rustup update
cargo update
rustc --version  # rustc 1.37.0 (eae3437df 2019-08-13)
cargo --version  # cargo 1.37.0 (9edd08916 2019-08-02)

The full error listing follows:

error: const fns are an unstable feature
  --> /Users/korymathewson/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.3.0/src/inline_lazy.rs:20:33
   |
20 |     pub const INIT: Self = Lazy(Cell::new(None), ONCE_INIT);
   |                                 ^^^^^^^^^^^^^^^
   |
   = help: in Nightly builds, add `#![feature(const_fn)]` to the crate attributes to enable

error[E0493]: constants are not allowed to have destructors
  --> /Users/korymathewson/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.3.0/src/inline_lazy.rs:20:33
   |
20 |     pub const INIT: Self = Lazy(Cell::new(None), ONCE_INIT);
   |                                 ^^^^^^^^^^^^^^^ constants cannot have destructors

error[E0493]: constants are not allowed to have destructors
  --> /Users/korymathewson/.cargo/registry/src/github.com-1ecc6299db9ec823/lazy_static-1.3.0/src/inline_lazy.rs:20:28
   |
20 |     pub const INIT: Self = Lazy(Cell::new(None), ONCE_INIT);
   |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ constants cannot have destructors

error: aborting due to 3 previous errors

error: Could not compile `lazy_static`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Then, attempted to recompile and got the following error:

cargo run --release
   Compiling parse-data-rs v0.1.0 (/Users/korymathewson/work/tv-show-ratings/parse-data-rs)
error[E0554]: #![feature] may not be used on the stable release channel
 --> src/main.rs:1:1
  |
1 | #![feature(nll)]
  | ^^^^^^^^^^^^^^^^

warning: unused `#[macro_use]` import
 --> src/main.rs:9:1
  |
9 | #[macro_use]
  | ^^^^^^^^^^^^
  |
  = note: #[warn(unused_imports)] on by default

warning: unused import: `Duration`
  --> src/main.rs:11:17
   |
11 | use std::time::{Duration, Instant};
   |                 ^^^^^^^^

warning: trait objects without an explicit `dyn` are deprecated
  --> src/main.rs:46:58
   |
46 | fn load_series() -> Result<Vec<ratings::db::Series>, Box<Error>> {
   |                                                          ^^^^^ help: use `dyn`: `dyn Error`
   |
   = note: #[warn(bare_trait_objects)] on by default

warning: trait objects without an explicit `dyn` are deprecated
   --> src/main.rs:141:21
    |
141 | ) -> Result<(), Box<Error>> {
    |                     ^^^^^ help: use `dyn`: `dyn Error`

warning: trait objects without an explicit `dyn` are deprecated
   --> src/main.rs:174:32
    |
174 | fn my_main() -> Result<(), Box<Error>> {
    |                                ^^^^^ help: use `dyn`: `dyn Error`

error[E0308]: mismatched types
  --> src/main.rs:61:27
   |
61 |             b"tvSeries" | b"tvMiniSeries" => {
   |                           ^^^^^^^^^^^^^^^ expected an array with a fixed size of 8 elements, found one with 12 elements
   |
   = note: expected type `&[u8; 8]`
              found type `&'static [u8; 12]`

error[E0308]: mismatched types
  --> src/main.rs:71:13
   |
71 |             b"tvEpisode" => {
   |             ^^^^^^^^^^^^ expected an array with a fixed size of 8 elements, found one with 9 elements
   |
   = note: expected type `&[u8; 8]`
              found type `&'static [u8; 9]`

error[E0277]: the trait bound `[u8]: std::convert::AsRef<[u8; 8]>` is not satisfied
  --> src/main.rs:60:30
   |
60 |         match data.titleType.as_ref() {
   |                              ^^^^^^ the trait `std::convert::AsRef<[u8; 8]>` is not implemented for `[u8]`
   |
   = help: the following implementations were found:
             <[T; 0] as std::convert::AsRef<[T]>>
             <[T; 10] as std::convert::AsRef<[T]>>
             <[T; 11] as std::convert::AsRef<[T]>>
             <[T; 12] as std::convert::AsRef<[T]>>
           and 31 others

warning: unused import: `rayon::prelude`
  --> src/main.rs:26:5
   |
26 | use rayon::prelude::*;
   |     ^^^^^^^^^^^^^^

error: aborting due to 4 previous errors

Some errors have detailed explanations: E0277, E0308, E0554.
For more information about an error, try `rustc --explain E0277`.
error: Could not compile `parse-data-rs`.

[phiresky]

Hey! Thanks for the detailed information. I've not seen those errors in your second log. It could be that compiling only works on Rust nightly (rustup install nightly && rustup default nightly), or that some dependencies need to be updated. I can probably check it out later today.

[korymath]

rustc 1.39.0-nightly (eceec57f7 2019-09-18)
cargo 1.39.0-nightly (9655d70af 2019-09-10)

Hmmm... doesn't seem to be fixing this all the errors:

cargo run --release
   Compiling cc v1.0.45
   Compiling libc v0.2.62
   Compiling proc-macro2 v0.4.30
   Compiling getrandom v0.1.12
   Compiling unicode-xid v0.1.0
   Compiling semver-parser v0.7.0
   Compiling syn v0.15.44
   Compiling byteorder v1.3.2
   Compiling cfg-if v0.1.9
   Compiling rustc-demangle v0.1.16
   Compiling ppv-lite86 v0.2.5
   Compiling either v1.5.3
   Compiling lazy_static v1.4.0
   Compiling memchr v2.2.1
   Compiling arrayvec v0.4.11
   Compiling autocfg v0.1.6
   Compiling typenum v1.11.2
   Compiling proc-macro2 v1.0.3
   Compiling log v0.4.8
   Compiling nodrop v0.1.13
   Compiling serde v1.0.101
   Compiling fixedbitset v0.1.9
   Compiling unicode-segmentation v1.3.0
   Compiling unicode-xid v0.2.0
   Compiling scopeguard v1.0.0
   Compiling remove_dir_all v0.5.2
   Compiling byte-tools v0.3.1
   Compiling regex-syntax v0.6.12
   Compiling multimap v0.4.0
   Compiling smallvec v0.6.10
   Compiling ryu v1.0.0
   Compiling rayon-core v1.6.0
   Compiling syn v1.0.5
   Compiling unicode-width v0.1.6
   Compiling opaque-debug v0.2.3
   Compiling itoa v0.4.4
   Compiling fake-simd v0.1.2
   Compiling semver v0.9.0
   Compiling itertools v0.8.0
   Compiling crossbeam-utils v0.6.6
   Compiling thread_local v0.3.6
   Compiling c2-chacha v0.2.2
   Compiling num-traits v0.2.8
   Compiling petgraph v0.4.13
   Compiling lock_api v0.3.1
   Compiling block-padding v0.1.4
   Compiling heck v0.3.1
   Compiling rustc_version v0.2.3
   Compiling crossbeam-queue v0.1.2
   Compiling backtrace-sys v0.1.31
   Compiling iovec v0.1.2
   Compiling num_cpus v1.10.1
   Compiling clicolors-control v1.0.1
   Compiling termios v0.3.1
   Compiling quote v0.6.13
   Compiling regex-automata v0.1.8
   Compiling memoffset v0.5.1
   Compiling parking_lot_core v0.6.2
   Compiling parking_lot v0.9.0
   Compiling quote v1.0.2
   Compiling generic-array v0.12.3
   Compiling rand_core v0.5.1
   Compiling bytes v0.4.12
   Compiling aho-corasick v0.7.6
   Compiling csv-core v0.1.6
   Compiling number_prefix v0.2.8
   Compiling digest v0.8.1
   Compiling block-buffer v0.7.3
   Compiling rand_chacha v0.2.1
   Compiling regex v1.3.1
   Compiling backtrace v0.3.37
   Compiling bstr v0.2.8
   Compiling serde_json v1.0.40
   Compiling sha2 v0.8.0
   Compiling rand v0.7.2
   Compiling crossbeam-epoch v0.7.2
   Compiling console v0.9.0
   Compiling failure v0.1.5
   Compiling csv v1.1.1
   Compiling tempfile v3.1.0
   Compiling crossbeam-deque v0.7.1
   Compiling indicatif v0.11.0
   Compiling which v2.0.1
   Compiling serde_derive v1.0.101
   Compiling prost-build v0.5.0
   Compiling prost-derive v0.5.0
   Compiling rayon v1.2.0
   Compiling prost v0.5.0
   Compiling prost-types v0.5.0
   Compiling parse-data-rs v0.1.0 (/Users/korymathewson/work/tv-show-ratings/parse-data-rs)
warning: unused `#[macro_use]` import
 --> src/main.rs:9:1
  |
9 | #[macro_use]
  | ^^^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused import: `Duration`
  --> src/main.rs:11:17
   |
11 | use std::time::{Duration, Instant};
   |                 ^^^^^^^^

warning: trait objects without an explicit `dyn` are deprecated
  --> src/main.rs:46:58
   |
46 | fn load_series() -> Result<Vec<ratings::db::Series>, Box<Error>> {
   |                                                          ^^^^^ help: use `dyn`: `dyn Error`
   |
   = note: `#[warn(bare_trait_objects)]` on by default

warning: trait objects without an explicit `dyn` are deprecated
   --> src/main.rs:141:21
    |
141 | ) -> Result<(), Box<Error>> {
    |                     ^^^^^ help: use `dyn`: `dyn Error`

warning: trait objects without an explicit `dyn` are deprecated
   --> src/main.rs:174:32
    |
174 | fn my_main() -> Result<(), Box<Error>> {
    |                                ^^^^^ help: use `dyn`: `dyn Error`

error[E0308]: mismatched types
  --> src/main.rs:61:27
   |
61 |             b"tvSeries" | b"tvMiniSeries" => {
   |                           ^^^^^^^^^^^^^^^ expected an array with a fixed size of 8 elements, found one with 12 elements
   |
   = note: expected type `&[u8; 8]`
              found type `&'static [u8; 12]`

error[E0308]: mismatched types
  --> src/main.rs:71:13
   |
71 |             b"tvEpisode" => {
   |             ^^^^^^^^^^^^ expected an array with a fixed size of 8 elements, found one with 9 elements
   |
   = note: expected type `&[u8; 8]`
              found type `&'static [u8; 9]`

error[E0277]: the trait bound `[u8]: std::convert::AsRef<[u8; 8]>` is not satisfied
  --> src/main.rs:60:30
   |
60 |         match data.titleType.as_ref() {
   |                              ^^^^^^ the trait `std::convert::AsRef<[u8; 8]>` is not implemented for `[u8]`
   |
   = help: the following implementations were found:
             <[T; _] as std::convert::AsRef<[T]>>
             <[T] as std::convert::AsRef<[T]>>
             <[u8] as std::convert::AsRef<bstr::bstr::BStr>>

warning: unused import: `rayon::prelude`
  --> src/main.rs:26:5
   |
26 | use rayon::prelude::*;
   |     ^^^^^^^^^^^^^^

error: aborting due to 3 previous errors

Some errors have detailed explanations: E0277, E0308.
For more information about an error, try `rustc --explain E0277`.
error: Could not compile `parse-data-rs`.

To learn more, run the command again with --verbose.

[phiresky]

I've improved the readme a bit, fixed rust compilation, and upgraded all dependencies to their newest versions. This fixes the above issues for me, let me know if it works for you.

[korymath]

Cargo now works ... and I can progress step-wise, but now I am stuck here:

 ~/work/tv-show-ratings   master ●  yarn run proto-gen
yarn run v1.6.0
$ mkdir lib && pbjs -t static-module -w es6 -o lib/ratings.js src/ratings.proto && pbts -o lib/ratings.d.ts lib/ratings.js
installing jsdoc@^3.5.5
installing tmp@0.0.33
installing uglify-js@^3.3.25
installing espree@^3.5.4
child_process.js:669
    throw err;
    ^

Error: Command failed: npm --silent install jsdoc@^3.5.5 tmp@0.0.33 uglify-js@^3.3.25 espree@^3.5.4
    at checkExecSyncError (child_process.js:629:11)
    at Object.execSync (child_process.js:666:13)
    at modInstall (/Users/korymathewson/work/tv-show-ratings/node_modules/protobufjs/cli/util.js:129:19)
    at Object.exports.setup (/Users/korymathewson/work/tv-show-ratings/node_modules/protobufjs/cli/util.js:156:5)
    at Object.<anonymous> (/Users/korymathewson/work/tv-show-ratings/node_modules/protobufjs/cli/pbjs.js:7:6)
    at Module._compile (internal/modules/cjs/loader.js:776:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[phiresky]

Can you say what the output of npm install jsdoc@^3.5.5 tmp@0.0.33 uglify-js@^3.3.25 espree@^3.5.4 in a empty directory is? Also, what is your node version? If it's not at least 10, upgrade it.

[korymath]

As expected it was the node version was out of date, so:

sudo npm cache clean -f
sudo npm install -g n
sudo n stable

Fixed things up, then your run code runs as expected once the permissions for the folders are correctly set:

# Restore ownership of the user's npm related folders, to the current user
# https://stackoverflow.com/questions/50639690/on-npm-install-unhandled-rejection-error-eacces-permission-denied
sudo chown -R $USER:$GROUP ~/.npm
sudo chown -R $USER:$GROUP ~/.config

 ✘  ~/tmtm  npm install jsdoc@^3.5.5 tmp@0.0.33 uglify-js@^3.3.25 espree@^3.5.4
npm WARN saveError ENOENT: no such file or directory, open '/Users/korymathewson/tmtm/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/Users/korymathewson/tmtm/package.json'
npm WARN tmtm No description
npm WARN tmtm No repository field.
npm WARN tmtm No README data
npm WARN tmtm No license field.

+ tmp@0.0.33
+ jsdoc@3.6.3
+ espree@3.5.4
+ uglify-js@3.6.0
added 34 packages from 142 contributors and audited 36 packages in 2.223s
found 0 vulnerabilities

Can you please confirm what the line: yarn run ts-node --transpile-only src/seed does? The status that it outputs is not very meaningful and I want to make sure that things are working correctly for each step. Should the seed be left running to serve the app?

[phiresky]

seed should output a magnet link like this:

It basically packs the data in parse-data-rs/data/ into a torrent file (this only needs to be done once per data update), and then it also starts a WebRTC (WebTorrent) peer that serves the data. Each user opening the website will also act as a server and fetch the data from other users. So for development you'll need to keep the seed script running, on the public website it will work without it as long as at least one user has the website open.

[korymath]

That is a cool and unique (to me) way of serving the data. I like it, thank you for introducing me to it. I’d love to read more about the benefits/theory motivating it if you have a pointer or two.

I’ll try to get mine up and running this afternoon given the help on package management.

On Fri, Sep 20, 2019 at 2:40 PM phiresky notifications@github.com wrote:

seed should output a magnet link like this:

[image: image] https://user-images.githubusercontent.com/2303841/65350553-464e2c00-dbe6-11e9-951f-7277fec95e5f.png

It basically packs the data in parse-data-rs/data/ into a torrent file (this only needs to be done once per data update), and then it also starts a WebRTC (WebTorrent) peer that serves the data. Each user opening the website will also act as a server and fetch the data from other users. So for development you'll need to keep the seed script running, on the public website it will work without it as long as at least one user has the website open.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/phiresky/tv-show-ratings/issues/8?email_source=notifications&email_token=AABLPMZBXSKNPLLLICWC3ODQKUKK5A5CNFSM4IYLEE4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7HRX6A#issuecomment-533666808, or mute the thread https://github.com/notifications/unsubscribe-auth/AABLPM4PTR7FXHUXQSC22JDQKUKK5ANCNFSM4IYLEE4A .

--

Kory Mathewson https://korymathewson.com

[phiresky]

The main motivation for me was that the license of using the data from IMDb was changed last year*, which means that actually hosting the data normally may have copyright issues now (that's why most similar websites that show tv show ratings got taken down). So this method means that both this repository as well as the hosted website do not contain any third party data and would in my opinion not be a valid target for a DCMA takedown request, since the data is distributed by the users themselves. It has the disadvantage of being very slow on initial load at least in some cases, since building peer to peer connections is really hard for many users (which shows the sad state of the internet in my opinion - the most basic functionality barely working because of NAT and firewalls that the users don't even control).

It's also an experiment to see how viable this method of distributing data is without users being bothered by it (or ideally even noticing it). PeerTube is similar. In my opinion, even interactive websites (with dynamic databases) could(should) be served this way in the future (with the addition of some consensus algorithm, pubkey cryptography, and a smart way to handle the dynamic data). ZeroNet does pretty much the same thing, but last I checked they didn't handle scalability at all (every user of a website downloads the whole history of all data).

I’d love to read more about the benefits/theory motivating it if you have a pointer or two.

I don't know of a good overview article that I'd say represents why I personally think this kind of thing is interesting / important, but in general I guess you could say it's part of the "re-decentralize the internet" "movement" for which you can find info about the motivation online.

---

_{* Considering the data is 100% user-contributed, and IMDb started out as a community project, I really don't see a moral issue with doing whatever the hell I want with it, even though Amazon has the legal ownership.}

[korymath]

This is a fantastic write-up and exactly what I was hoping for. It was one of the reasons that this project interested me and I am happy to see that this is what is behind the architecture...

Bit of a topical shift, but what would be your opinion of using something like: https://ipfs.io/ for this?

[phiresky]

I like ipfs, but the main ipfs implementation itself doesn't work within the browser, so to use it the website would need to directly depend on one ipfs "entry" node (basically just pull the data from https://ipfs.io/ipfs/* or https://cloudflare-ipfs.com/ipfs/* ), which I don't like. I know there's work going on on a pure js ipfs implementation (also using WebRTC) at https://js.ipfs.io, but I'm not sure what the state is. If that works, it would be pretty ideal for this project, yeah. Might also scale better than WebTorrent with regards to finding peers and fetching data from only exactly one TV show from a large list of shows.

Also, and I'm not sure if they fixed it yet, but for many of these decentralized protocols where in-browser was not a main priority from the start, the browser clients can't actually exchange data with the native versions. e.g. go-ipfs instances can't communicate with any js-ipfs instances, and WebTorrent peers can't share data with normal torrent peers. The main reason for this is that apparently the WebRTC protocol is complex enough that there really only exists one "embeddable" implementation: The one in Chromium. So to support WebRTC, you need to pull in a huge dependency, which e.g. normal torrent clients like qbittorrent would probably never do. For example the seed.ts script indirectly pulled in a whole Electron/Chrome instance just to support WebRTC until a while ago. This isn't a problem for this project in particular, but it's one of the reasons why a lot of related stuff needs native software installed somewhere at some point (like the DAT project which doesn't even try to integrate with browsers).

[korymath]

Ahhh. That makes sense. I understand this limitation much better now. Thank you for taking the time to answer these questions in this forum. It is enlightening. A testament to the strength of the open-source community.

[korymath]

Alright, things are moving along for the installation... but now I am not able to serve the data from server to client-mesh. The following happens with the torrent server running in a terminal locally, and multiple open windows to the site... Perhaps I am still missing a step on server launch?

# javacript console.

got no peers from tracker
index.js:59 WebSocket connection to 'wss://tracker.btorrent.xyz/' failed: Error during WebSocket handshake: Unexpected response code: 522
Socket @ index.js:59
load.ts:35 Error: connection error to wss://tracker.btorrent.xyz
    at WebSocket.Socket._ws.onerror (index.js:78)

[phiresky]

Sorry for the late reply, I missed your response :/

The connection to tracker.btorrent.xyz failing is expected since that tracker has been down for a while.

Can you check whether your the seed script is serving the same torrent as the browser is trying to load? The seed script should be outputting something like magnet:?xt=urn:btih:df8e0ada7a1d4380ee92a37a02e8ea580139a5cb

and the browser console should show infoHash df8e0ada7a1d4380ee92a37a02e8ea580139a5cb. Note the hash is the same. If not, make sure you run yarn run dev after starting and waiting for the output of yarn run ts-node --transpile-only src/seed. If that doesn't help, trying running rm -rf .cache and then run seed and dev again since parcel's caching is sometimes buggy.

[korymath]

Bingo. Thank you for the incredible support and development/deployment effort. I commend you on a stellar/thoughtful open-source project. I enjoyed your discussion on how you might deliver this kind of data.

[phiresky]

Thank you for your kind words. Always happy to help :)