Using Firestore as a Front End Database

[TimeTravelersHackedMe]

I noticed that this software bashes the server with a lot of API calls and I've also noticed some sites experiencing downtime on their front end (likely because of all the calls). I'm in the process of re-writing an Ionic app to replace PoolUI. Part of the project entails writing another pm2 process that will push data to Firestore. I'd like to have it be as real-time as possible and I'm also trying to avoid making unnecessary pushes to Firestore. My question is about the Blocks and Payment page.. Do the entries update after they're posted? In other words, will the blocks or payments on the fifth page in pagination change in data? If not, I don't have to cycle through the whole list and I'll be able to update Firestore at smaller intervals.

Here's the TypeScript I have so far. This code is working, just posting it here just in case anyone else is interested in this:

'use strict';

import * as firebase from 'firebase-admin';
import * as request from 'request';
import * as cron from 'cron';

const key = require('../X.json');
firebase.initializeApp({
    credential: firebase.credential.cert(key),
    databaseURL: 'X'
});
const db = firebase.firestore();

const baseUrl = 'https://monero.gives/api';
const config = {
    url: {
        poolStats: baseUrl + '/pool/stats',
        networkStats: baseUrl + '/network/stats',
        config: baseUrl + '/config',
        ports: baseUrl + '/pool/ports'
    }
}

interface PoolStats {
    last_payment: number,
    pool_list: Array<string>,
    pool_statistics: {
        hashRate: number,
        lastBlockFound: number,
        lastBlockFoundTime: number,
        miners: number,
        roundHashes: number,
        totalBlocksFound: number,
        totalHashes: number,
        totalMinersPaid: number,
        totalPayments: number
    }
}

interface NetworkStats {
    difficulty: number,
    hash: string,
    height: number,
    ts: number,
    value: number
}

interface PoolConfig {
    pplns_fee: number,
    pps_fee: number,
    solo_fee: number,
    btc_fee: number,
    min_wallet_payout: number,
    min_btc_payout: number,
    min_exchange_payout: number,
    dev_donation: number,
    pool_dev_donation: number,
    maturity_depth: number,
    min_denom: number
}

interface PortsConfig {
    global: Array<PortConfig>,
    pplns: Array<PortConfig>
}

interface PortConfig {
    host: {
        ip?: string,
        blockID: number,
        blockIDTime: number,
        hostname: string
    },
    port: number,
    pool_type: string,
    difficulty: number,
    miners: number,
    description: string,
}

function get(url): Promise<any> {
    return new Promise((resolve, reject) => {
        const req = request.get(url)
            .on('response', function (response) {
                if (response.statusCode === 200) {
                    req.on('data', (data) => {
                        const json = JSON.parse(data);
                        return resolve(json);
                    });
                } else {
                    return reject(new Error('Request returned a status code that was ' + response.statusCode));
                }
            })
            .on('error', (e) => {
                return reject(new Error('Failed to request ' + url));
            });
    });
}

class Sync {
    static historyCount: number = 0;

    static async poolStats() {
        try {
            const pool: PoolStats = await get(config.url.poolStats);
            const stats = {
                lastPayment: pool.last_payment,
                poolList: pool.pool_list,
                hashRate: pool.pool_statistics.hashRate,
                lastBlockFound: pool.pool_statistics.lastBlockFound,
                lastBlockFoundTime: pool.pool_statistics.lastBlockFoundTime,
                miners: pool.pool_statistics.miners,
                roundHashes: pool.pool_statistics.roundHashes,
                totalBlocksFound: pool.pool_statistics.totalBlocksFound,
                totalHashes: pool.pool_statistics.totalHashes,
                totalMinersPaid: pool.pool_statistics.totalMinersPaid,
                totalPayments: pool.pool_statistics.totalPayments,
                updateTime: new Date().getTime(),
                historyCount: this.historyCount
            }
            return Promise.all([
                await db.collection('pool').doc('stats').update(stats),
                await db.collection('pool').doc('stats').collection('history').doc(new Date().getTime().toString()).set(stats)
            ]);
        } catch (e) {
            throw new Error('Sync.poolStats() error: ' + e);
        }
    }

    static async networkStats() {
        try {
            const network: NetworkStats = await get(config.url.networkStats);
            const stats = {
                difficulty: network.difficulty,
                hash: network.hash,
                height: network.height,
                timestamp: network.ts,
                value: network.value,
                updateTime: new Date().getTime(),
                historyCount: this.historyCount
            }
            return await Promise.all([
                await db.collection('network').doc('stats').update(stats),
                await db.collection('network').doc('stats').collection('history').doc(new Date().getTime().toString()).set(stats)
            ]);
        } catch (e) {
            throw new Error('Sync.networkStats(): ' + e);
        }
    }

    static async config() {
        try {
            const configs: PoolConfig = await get(config.url.config);
            const stats = {
                pplnsFee: configs.pplns_fee,
                ppsFee: configs.pps_fee,
                soloFee: configs.solo_fee,
                btcFee: configs.btc_fee,
                minWalletPayment: configs.min_wallet_payout,
                minBtcPayout: configs.min_btc_payout,
                minExchangePayout: configs.min_exchange_payout,
                devDonation: configs.dev_donation,
                poolDevDonation: configs.pool_dev_donation,
                maturityDepth: configs.maturity_depth,
                minDenom: configs.min_denom
            }
            return await db.collection('pool').doc('config').update(stats);
        } catch (e) {
            throw new Error('Sync.config(): ' + e);
        }
    }

    static async ports() {
        try {
            const ports: PortsConfig = await get(config.url.ports);
            const promises = [];
            for (const setting of ports.global) {
                promises.push(await db.collection('pool').doc('ports').collection('global').doc(setting.host.hostname + ':' + setting.port).set(formatPortSetting(setting, 'global')));
            }
            for (const setting of ports.pplns) {
                promises.push(await db.collection('pool').doc('ports').collection('pplns').doc(setting.host.hostname + ':' + setting.port).set(formatPortSetting(setting, 'pplns')));
            }
            return await Promise.all(promises);
        } catch (e) {
            throw new Error('Sync.ports(): ' + e);
        }
    }
}

function formatPortSetting(data: PortConfig, poolType: string) {
    const json: any = {
        blockId: data.host.blockID,
        blockIdTime: data.host.blockIDTime,
        hostname: data.host.hostname,
        port: data.port,
        poolType: poolType,
        difficulty: data.difficulty,
        miners: data.miners,
        description: data.description
    }
    /*if (data.host.ip !== 'undefined') {
        json.ip = data.host.ip;
    }*/
    return json;
}

async function runStatsJob() {
    try {
        await Promise.all([
            await Sync.poolStats(),
            await Sync.networkStats()
        ]);
        Sync.historyCount === 100 ? Sync.historyCount = 0 : Sync.historyCount++;
    } catch (e) {
        throw new Error('runeStatsJob() Promise.all: ' + e);
    }
}

new cron.CronJob({
    cronTime: '* * * * *',
    onTick: async () => {
        try {
            setTimeout(async () => {
                await runStatsJob();
            }, 30000);
            await runStatsJob();
        } catch (e) {
            throw new Error('runStatsJob() cron: ' + e);
        }
    },
    start: true
});

new cron.CronJob({
    cronTime: '* * * * *',
    onTick: async () => {
        try {
            await Promise.all([
                await Sync.config(),
                await Sync.ports()
            ]);
        } catch (e) {
            throw new Error('config()/ports() cron: ' + e);
        }
    },
    start: true
});

The historyCount static variable is used so I can index the network/pool hash rates while still updating them regularly. This is used so I can display charts for them on the front end without pulling a ridiculous amount of data. The pool stats are updating every 30 seconds (along with network stats) and the pool config is updating every 5 minutes in this example.

Also, are there any features you're looking to bring to the front end? I'm a professional front end engineer who teaches at a university as well. I'm currently re-writing PoolUI in Ionic so we're using Angular 5 and can port the interface to Android/iOS apps/(Electron with a built-in miner). I'm also interested in adding a browser miner but I haven't done too much research on that yet.

[bobbieltd]

Very nice poolUI, I like it 👍👍👍. Great job ! For api and databse, I’m noob and can’t help.

[TimeTravelersHackedMe]

Bump. Anyone know if the Payments/Blocks page's historical data gets updated?

[camthegeek]

Typically, it is done by calling the pool's api to /pool/payments?page=#&limit=#

Example, page=1&limit=100 https://api.aeonminingpool.com/api/pool/payments?page=1&limit=100

Results in last 100 payments on page 1. Same method with blocks.

Make sense?

[TimeTravelersHackedMe]

Yeah... I know that... I'm wondering if the historical data is updated. I want to know if I can cache older values so the database doesn't have to query anything if it is already cached.

[camthegeek]

The values of previous payments and blocks should never change.