Tweetscape: The Supercharged Twitter Feed

Tweetscape surfaces the best "insider" information—and the conversation around it—as shared by the smartest people in a given topic (e.g. ETH, BTC, NFTs, or Tesla) on Twitter. Tweetscape curates article links shared by the most reputable accounts on Twitter for a number of topics (e.g. ETH, BTC, NFTs, or Tesla). Learn more here.

How it works

High level

Tweetscape uses hive.one to determine who are the most reputable (i.e. the "smartest") people in a specific field (e.g. who are the experts in ETH, BTC, NFTs, or Tesla) on Twitter; hive.one acts as a reputation layer for the internet, determining who you can trust through a weighted graph of who follows who (e.g. a reputable user following another user raises that other user's "attention score" by more than if some random Joe follows them).

Tweetscape then uses Twitter's API and that list of "smartest" people to get links to the articles most abundantly (and most recently) shared by the "smartest" people on Twitter for a given topic (e.g. ETH, BTC, NFTs, or Tesla). It also shows you the conversation around each link; you get to see the best links and what the smartest people are saying about them.

Low level

Tweetscape is a full-stack React application built with Remix and deployed on Fly.

Contributing

Project Structure

This repository is a React app built in Typescript with Remix, TailwindCSS, and PostgreSQL. Both the Remix app and PostgreSQL database are deployed worldwide as edge-based, clustered applications on Fly:

.
├── app
│   ├── cookies.server.ts
│   ├── db.server.ts
│   ├── entry.client.tsx
│   ├── entry.server.tsx
│   ├── img.server.ts
│   ├── root.tsx
│   ├── routes
│   │   ├── $cluster.tsx
│   │   ├── img.$url.ts
│   │   └── index.tsx
│   ├── styles
│   │   └── app.css
│   └── utils.server.ts
├── db
│   ├── copy.pgsql
│   └── setup.pgsql
├── Dockerfile
├── fly.toml
├── LICENSE
├── package.json
├── postcss.config.js
├── public
│   ├── favicon.ico
│   ├── fonts
│   │   ├── sans.css
│   │   ├── sans.ttf
│   │   ├── sans.woff
│   │   ├── sans.woff2
│   │   ├── serif.css
│   │   └── serif-vietnamese.woff2
│   └── pics
│       ├── placeholder.png
│       └── vanessa.jpg
├── README.md
├── remix.config.js
├── remix.env.d.ts
├── scripts
│   ├── data.mjs
│   └── utils.mjs
├── styles
│   └── app.css
├── tailwind.config.js
├── tsconfig.json
├── wrangler.toml
└── yarn.lock

10 directories, 66 files

The app/ directory contains all of the front-end Remix components, server-side business logic (files suffixed with .server.ts will only be included in the server-side bundle), and pages.
The public/ directory contains publicly visible assets (e.g. fonts, pictures, favicons, etc).
The scripts/ directory defines a set of Node.js scripts used to seed our PostgreSQL database with data from Twitter's and Hive's APIs.
The db/ directory contains .pgsql scripts that define our database schema. Use them with care (i.e. don't accidentally reset the production database).
The styles/ directory contains the entry point for the TailwindCSS compiler (which outputs app/styles/app.css which you shouldn't ever edit manually).
And, of course, the root directory contains a number of configuration files (for ESLint, Prettier, Tailwind, Fly, a Dockerfile for Fly deployments, etc).

Database Types

I could've used an ORM like Prisma to automatically generate Typescript type definitions and my PostgreSQL database schema at the same time, but I like being able to use the full power of both languages separately, so I've redefined my data types twice: once in SQL (db/setup.pgsql) and once in Typescript (app/db.server.ts).

The data structures that I chose and the property names that I chose come directly from Twitter's API and Hive's API. I aimed to be as unopinionated as possible when storing their data; I stored everything that was returned in—what I think is—a perfectly normalized PostgreSQL schema.

It can be useful to install something like Beekeeper Studio to open the PostgreSQL database in a visual spreadsheet-like form in order to better understand what each table actually contains and how they relate to one another.

Development Environment

To setup the development environment for and to contribute to Tweetscape:

Follow these instructions to install nvm (our suggested way to use Node.js) on your machine. Verify that nvm is installed by running:

$ command -v nvm

Run the following command to install Node.js v16.14.0 (the current LTS that's specified in our .nvmrc):

$ nvm use

(Optional) Run the following command to set Node.js v16.14.0 as your default Node.js version (useful if you have multiple Node.js versions installed and don't want to have to remember to switch to v16.14.0):

$ nvm alias default 16.14.0

Ensure that you have recent versions of Node.js and it's package manager npm by running:

$ node -v
16.14.0
$ npm -v
8.3.1

(Optional) Install the Cypress system dependencies if you plan on running our integration tests locally.

$ sudo apt-get install libgtk2.0-0 libgtk-3-0 libgbm-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth xvfb

Clone and cd into this repository locally by running:

$ git clone https://github.com/rooteco/tweetscape.git && cd tweetscape/

Follow these instructions to install yarn (our dependency manager for a number of reasons):

$ corepack enable

Then, install of our project's dependencies with the following command:

$ yarn

Copy over the .env file (which contains project secrets) from this private repository:

$ cp tweetscape-env/.env* tweetscape/

Install the flyctl CLI, login to the CLI, and setup port forwarding so you can access the PostgreSQL database and Redis instance from your machine:

$ fly proxy 5432 -a tweetscape-pg
$ fly proxy 6379 -a tweetscape-redis

(Optional) Install Beekeeper Studio to be able to visually manipulate the PostgreSQL data. Or, you should be able to simply access the psql command interface directly:

$ psql postgres://postgres:<pwd-from-env-file>@localhost:5432/postgres

Finally, you should be able to start a fully-functioning development server:

$ yarn dev

(Optional) To run fully local versions of our deployed infrastructure, you'll have to first make sure you've got Docker properly installed and configured on your system. Then, you'll be able to start a local PostgreSQL database and Redis instance by running:

$ yarn concurrently yarn:dev:postgres yarn:dev:redis

Commit Message Format

I have very precise rules over how Git commit messages must be formatted. This format leads to easier to read commit history. Please refer to the following documentation for more info:

Commit Message Header

Commit messages that do not adhere to the following commit style will not be merged into develop:

<type>(<scope>): <short summary>
  │       │             │
  │       │             └─⫸ Summary in present tense. Not capitalized. No period at the end.
  │       │
  │       └─⫸ Commit Scope: The page, API route, or component modified.
  │
  └─⫸ Commit Type: ci|docs|feat|fix|perf|refactor|test|deps|chore

The <type> and <summary> fields are mandatory, the (<scope>) field is optional.

Type

Must be one of the following:

ci: Changes to our CI configuration files and scripts.
docs: Documentation only changes.
feat: A new feature.
fix: A bug fix.
perf: A code change that improves performance.
refactor: A code change that neither fixes a bug nor adds a feature.
test: Adding missing tests or correcting existing tests.
deps: A change in dependencies.
chore: A code change in utility scripts, build configurations, etc.

Scope

The scope should refer to the page, API route, or component modified. This can be flexible however (e.g. the scope for a docs: commit may be the README).

Summary

Use the summary field to provide a succinct description of the change:

Use the imperative, present tense: "change" not "changed" nor "changes".
Don't capitalize the first letter.
No dot (.) at the end.

rooteco / tweetscape

readme