search

redis-developer / redis-hacker-news-demo

The demo is a Hacker News clone and uses RedisJSON for storing the data and RediSearch for searching.
MIT License
16 stars 10 forks source link

readme

Tutorial: Hacker News clone that uses Redis as a primary database (Redis JSON + Redis Search)

Technical Stack

Database Schema

User

username: STRING  // unique id of user
password: STRING // hashed password of user
authToken: STRING // cookie token
authTokenExpiration: STRING
resetPasswordToken: STRING
resetPasswordTokenExpiration: STRING
email: STRING // email of user
created: NUMBER // timestamp of user createdAt
karma: NUMBER // karma of user
about: STRING
showDead: BOOLEAN // indicates if user account is dead
isModerator: BOOLEAN // indicates if user is moderator
shadowBanned: BOOLEAN // indicates if user account is shadow banned
banned: BOOLEAN // indicates if user account is banned

Item

This is the item posted by user. I can be a link or post with some description.

id: STRING // id of item
by: STRING // username who posted this item
title: STRING // title of item
type: STRING // 'show' | 'news' | 'ask'
url: STRING // link when type is news or show
domain: STRING // domain of url
text: STRING // description when type is ask
points: NUMBER // points of item
score: NUMBER // score of item
commentCount: NUMBER // total comment counts
created: NUMBER // created timestamp
dead: BOOLEAN // is killed by moderator

Comment

id: STRING
by: STRING // username of who posed this comment
parentItemId: STRING // id of parent item
parentItemTitle: STRING // title of parent item
isParent: BOOLEAN // if this is root comment of a item
parentCommentId: : STRING // if it's not root, indicate the parent comment id
children: ARRAY // id of array of children comments
text: STRING // comment content
points: NUMBER
created: NUMBER
dead: BOOLEAN

UserFavorite

username: STRING
type: STRING  // 'comment' | 'item'
id: STRING // id of that comment or item
date: NUMBER

UserVote, UserHidden

The schema is exact same as UserFavorite but indicates if user has voted on item/comment, if user has marked hidden for a specific item/comment

ModeratorLog

This schema is to store and keep track of moderator action history.

How document and each data type is stored in Redis.

How do you store a document?

We want to store a document(like a user) in redis. Basically, indexable and sortable fields are stored in hash while we store rest of the fields in Redis JSON. We can applyRedis Searchqueries once we store in hash.

2 databases were in redislabs, one withRedis Searchenabled and the other one with Redis JSON enabled. Example user document

{
    username: 'andyr',
    password: '$2a$10$W4UUtt5hkoiDtKU1.ZR6H.EklDH1ePUpZTvEI2IBrYRrufx8WMvIO',
    authToken: 'EklDH1ePUpZTvEI2IBrYRrufx8WMvIO',
    created: 1541648957,
    isModerator: false,
}

We need to search by username, created, isModerator but we dont need to search by password and authToken. Thus, username, created and isModerator will be saved in Redis Search. password and authToken will be saved in Redis JSON.

Store indexable/sortable fields in hash ofRedis Searchdb

HSET user:1 username andyr created 1541648957 isModerator false

Store other fields in Redis JSON

JSON.SET user:1 . '{ "password": "$2a$10$W4UUtt5hkoiDtKU1.ZR6H.EklDH1ePUpZTvEI2IBrYRrufx8WMvIO", "authToken": "EklDH1ePUpZTvEI2IBrYRrufx8WMvIO" }'

Get user where username is 'andyr'

Managing id of a new document

When a new document added, we need to know the unique key to store value. {collectionName}:id-indicator indicates the next unique number to use as key. It's increased whenever a new document is creaed and that id is used.

How to update a document?

We need to update isModerator to true from above user document. If it's indexable field,

HSET user:1 isModerator true

If not,

JSON.SET user:1 isModerator true

How to delete a document?

Just delete the keys on both dbs.

DEL user:1

DataTypes

  1. STRING - Stored as plain string format in hash key and in Redis JSON
  2. NUMBER - Same as STRING
  3. ARRAY - It won't be stored in hash, it will be only stored in Redis JSON
  4. BOOLEAN - It can be stored as STRING, and apply queries
  5. DATE - It can be converted into timestamp, and application will use the single timezone when it's stored in db.

User Schema

{
  id: {
    type: RedisDbTypes.STRING,
    indexed: true,
  },
  by: {
    type: RedisDbTypes.STRING,
    indexed: true,
  },
  parentItemId: {
    type: RedisDbTypes.STRING,
    indexed: true,
  },
  parentItemTitle: {
    type: RedisDbTypes.STRING,
    indexed: true,
    sortable: true,
  },
  isParent: {
    type: RedisDbTypes.BOOLEAN,
    indexed: true,
  },
  parentCommentId: {
    type: RedisDbTypes.STRING,
    indexed: true,
  },
  children: {
    type: RedisDbTypes.ARRAY,
    refIdx: 'comment',
    default: [],
  },
  text: {
    type: RedisDbTypes.STRING,
    indexed: true,
  },
  points: {
    type: RedisDbTypes.NUMBER,
    default: 1,
    min: -4,

    indexed: true,
    sortable: true,
  },
  created: {
    type: RedisDbTypes.NUMBER,

    indexed: true,
    sortable: true,
  },
  dead: {
    type: RedisDbTypes.BOOLEAN,
    default: false,
    indexed: true,
  }
}

Creating Indices

When schema is created, it should check db, and make sure indexable/sortable fields are configured properly. To check if indcies are configured properly.

FT.INFO idx:user

If one of the field is not configured, it should reconfigureRedis Searchfor example

FT.CREATE idx:user ON hash PREFIX 1 user: SCHEMA username TEXT SORTABLE email TEXT SORTABLE created NUMERIC SORTABLE karma NUMERIC SORTABLE about TEXT showDead TEXT isModerator TEXT shadowBanned TEXT banned TEXT _id TEXT SORTABLE

Please note that how each field are indexed and marked as sortable.

Example Key/Values of each schema

How it works

By Screens

Signup

Signup Screen

Login

Login Screen

Item list page

Newest Screen

Submit

Submit Screen

Update Profile

Update Profile Screen

Search

Search Screen

Example commands

There are 2 type of fields, indexed and non-indexed.

  1. Indexed fields will be stored in hash using HSET/HGET.
  2. Non-indexed fields will be stored in Redis JSON.

Pull Hacker News API to seed database

Using API, it pulls the latest hackernews data.

Running Locally

.env

Copy .env.sample to .env and provide the values.

Start the dev server

npm run dev

Start the production server

npm run build
npm start