search

EnCiv / undebate

Not debates, but recorded online video Q&A with candidates so voters can quickly can get to know them, for every candidate, for every election, across the US.
Other
20 stars 14 forks source link

Real-time SMPreview Generation #237

Open ddfridley opened 4 years ago

ddfridley commented 4 years ago

As is, smpreveiw will generate preview images for all the new viewers in the database, once a day. Now we need a way to generate new preveiws for viewers as soon as there is a new recording.

in https://github.com/EnCiv/undebate/tree/master/app/server/events it's possible to create an event handler similar to notify-of-new-participant.js to send the new iota to the smpreview server to trigger generating a new preview image.

How we connect from the undebate server to the smpreview server is up for investigation. Could be an https request, could be socket.io

Things that run on heroku tend to need an https server, socket.io will start a server, or we could just run an express server to accept requestions.

Also- please put in some level of security so people can't randomly probe the smpreview server and cause things to happen. I do see people probing the servers common paths that have vulnerabilities.

djbowers commented 4 years ago

or we could just run an express server to accept requestions.

What's a requestion? 😆

ddfridley commented 4 years ago

Interesting slip. Should have been requests.

ddfridley commented 4 years ago

For the Socket IO implementation idea this is a file from a private repo for the socketAPI called socket-api-server.js. If you use this, it might make sense to turn this into a separate repo, with a README file.

"use strict";

const serverIo = require("socket.io");

const api_keys=JSON.parse(process.env.SOCKET_API_KEYS|| "[]");

function fromKeyList(packet){
    let index=api_keys.indexOf(packet[1]);
    logger.info("fromKeyList found:",index)
    return index > -1;
}

var initialized=false;

function socketAPIServer(apis, initFunction, authenticator=fromKeyList){
    const server = serverIo.listen(process.env.PORT || 8000);
    server.on("connection", (socket) => {
        if(!initialized) {
            initialized=true; 
            process.nextTick(initFunction);
        }
        // this must be first - to block unauthenticated access to the APIs
        socket.use((packet,next)=>{
            if(socket.auth) return next();
            else if(packet[0]==="authenticate" && authenticator(packet)){ 
                socket.auth=true;
                socket.emit("authenticated")
                return next();
            }
            else return next(); // silently ignore unauthorized packets
        })
        socket.on("disconnect", () => {
            delete socket.auth;
        });
        socket.on("force-disconnect", (...args) => {
            console.info("socketAPIServer received force-disconnect",...args);
            delete socket.auth;
            if(args[0]==='exit') {
                console.error("not an error but disconnect exit received. exiting process")
                process.nextTick(()=>process.exit());
            }
        });
        apis.forEach(api=>{
            socket.on(api.name,(...args)=>{
                if(!socket.auth) return;  // silently ignore unauthorized packets
                try {
                    api.func(...args)
                }
                catch(error){
                    console.error("socketAPIServer caught error from api",api.name,error);
                }
            })
        })
    });
    return server;
}

module.exports=socketAPIServer;

Then this is an example of code that uses the socket-api-server. The api calls like "candidate_videos", and their corresponding function would be specific to the application.

dataIsReady is a var that is set by main() after the server is ready to process api calls. Like after all the mongo database stuff has been read in and prepared. There might be a better way to do this - but it worked for now.

var dataIsReady = false;
var queued = [];

const APIs = [
    {
        name: "candidate_videos", func: (year, cb) => {
            function reply() {
                cb(Object.values(encivInfo.recorders).reduce((acc,recorder)=>{
                    const csrId=recorder.bp_info && recorder.bp_info.candidate_stage_result_id;
                    if(bpInfo.candidateStageResults[csrId])
                        acc.push({candidate_stage_result: csrId, 
                            recorder_url: CC_HOST + recorder.path,
                            recorded: bpInfo.candidateStageResults[csrId].recorded || false
                        })
                    return acc;
                },[]))
                logger.info("candidate_videos api reply sent")
            }
            if (!dataIsReady) {
                queued.push(reply)
            } else
                reply();
        }
    },
    {
        name: "stage_videos", func: (year, cb) => {
            function reply() {
                cb(Object.values(encivInfo.viewers).reduce((acc,viewer)=>{
                    const stageId=viewer.bp_info && viewer.bp_info.stage_id;
                    if(!bpInfo.stages[stageId]) {
                        logger.error("stage_videos, no stage found:",stageId,"for", viewer.path, "skipping.");
                        return acc;
                    }
                    const {cc_recorded}= bpInfo.stages[stageId];
                    if(cc_recorded) acc.push({stage: stageId, viewer_url: CC_HOST +viewer.path})
                    return acc;
                }, []))
                logger.info("stage_videos api reply sent")
            }
            if (!dataIsReady) {
                queued.push(reply)
            } else
                reply();
        }
    }
];

async function main(){
   await ... initialize stuff
   dataIsReady=true;  // there is probably a better way to do this with a promise
}

var server;
try {
    server=socketAPIServer(APIs,main); // main is the function that should run after the API server is started
}

Then this file setups up the socket on a client

https://github.com/EnCiv/get-enciv-info/blob/master/index.js

And this file makes API calls on a client: https://github.com/EnCiv/get-enciv-info/blob/master/demo.js

djbowers commented 4 years ago

@ddfridley I pushed my code so far to branches on my forks for each of the projects. It's not fully working or done yet though, just fair warning.

The socket-api code is on the master branch of https://github.com/djbowers/socket-api Undebate code is on branch realtime-smpreview of https://github.com/djbowers/undebate/tree/realtime-smpreview SMPreview code is also branch realtime-smpreview of https://github.com/djbowers/smpreview/tree/realtime-smpreview

If you have some time to take a look, feel free. Otherwise maybe let's sync up on Monday to chat about it? We can go over the code, and I have a few questions we can talk about. I'll list them below as well.

  1. Should the undebate server maintain a socket.io connection with the smpreview server at all times? Or should it start up the connection, make the api call, and then disconnect? Right now, it's disconnecting too early, and I'm just trying to figure out the best way to fix it.
ddfridley commented 4 years ago

Some trouble communications from on server to another. Need to move the the socketapi to the new repo.

djbowers commented 4 years ago

The PR's are now all open and linked together. They are still drafts until I get things working on Heroku. My current suspicion is that Heroku doesn't like the socket.io server without it being connected to an http server, at least the way it's currently configured. Any thoughts here @ddfridley ?

Once the system is working on Heroku I will publish the PR's.

Edit: Heroku docs for using socket.io with Node.js recommend using an http server https://devcenter.heroku.com/articles/node-websockets#option-2-socket-io

I suspect they need the http server to put the dynos to sleep? I will investigate further.

djbowers commented 4 years ago

Ok I got everything working properly on Heroku.

I determined that if you don't specify your own http server, socket.io does it for you automatically under the hood. So when you specify the SOCKET_API_URL, you still have to use http://

So I was correct in that I had my URL's wrong on Wednesday.

Also apparently I had my smpreview server on "maintenance mode" on Heroku. Not sure how that happened, I don't remember turning it on.

Anyway, here is what the console output looks like on each server when a real-time preview is generated:

smpreview server / socket-api server

2020-08-29T00:35:54.432943+00:00 heroku[router]: at=info method=GET path="/socket.io/?EIO=3&transport=polling&t=NGtf6EH&b64=1" host=undebate-smpreview.herokuapp.com request_id=34a46880-6fe4-454c-8ac2-aa640b994248 fwd="3.88.33.78" dyno=web.1 connect=1ms service=7ms status=200 bytes=344 protocol=http
2020-08-29T00:35:54.466702+00:00 heroku[router]: at=info method=GET path="/socket.io/?EIO=3&transport=polling&t=NGtf6FF&b64=1&sid=ryib_nnCpOFI0GieAAAA" host=undebate-smpreview.herokuapp.com request_id=65113728-8ba5-4211-8d63-d27707b2c57c fwd="3.88.33.78" dyno=web.1 connect=0ms service=6ms status=200 bytes=262 protocol=http
2020-08-29T00:35:54.467666+00:00 heroku[router]: at=info method=POST path="/socket.io/?EIO=3&transport=polling&t=NGtf6FC&b64=1&sid=ryib_nnCpOFI0GieAAAA" host=undebate-smpreview.herokuapp.com request_id=7e131abf-b9d1-4e76-98d6-e5d62048466b fwd="3.88.33.78" dyno=web.1 connect=0ms service=4ms status=200 bytes=225 protocol=http
2020-08-29T00:35:54.493096+00:00 heroku[router]: at=info method=POST path="/socket.io/?EIO=3&transport=polling&t=NGtf6Fc&b64=1&sid=ryib_nnCpOFI0GieAAAA" host=undebate-smpreview.herokuapp.com request_id=c912aeb0-894c-40d9-9f1e-8090e52d8fe3 fwd="3.88.33.78" dyno=web.1 connect=0ms service=2ms status=200 bytes=225 protocol=http
2020-08-29T00:35:54.434064+00:00 app[web.1]: connected to socket: ryib_nnCpOFI0GieAAAA
2020-08-29T00:35:54.465449+00:00 app[web.1]: fromKeyList found: 1
2020-08-29T00:35:54.493139+00:00 app[web.1]: calling generate_smpreview with args: {
2020-08-29T00:35:54.493140+00:00 app[web.1]:   parentId: '5f2b38317f20777395c58939',
2020-08-29T00:35:54.493141+00:00 app[web.1]:   subject: 'Participant:realtime-smpreview-Candidate Recorder',
2020-08-29T00:35:54.493142+00:00 app[web.1]:   description: 'A participant in the following discussion:A Candidate Recorder for the Conversation: realtime-smpreview',
2020-08-29T00:35:54.493142+00:00 app[web.1]:   component: {
2020-08-29T00:35:54.493143+00:00 app[web.1]:     component: 'MergeParticipants',
2020-08-29T00:35:54.493143+00:00 app[web.1]:     participant: {
2020-08-29T00:35:54.493143+00:00 app[web.1]:       speaking: [Array],
2020-08-29T00:35:54.493144+00:00 app[web.1]:       name: 'Test Heroku 4',
2020-08-29T00:35:54.493146+00:00 app[web.1]:       listening: 'https://res.cloudinary.com/hrewc5ehd/video/upload/q_auto/v1598661353/5ebda7b58fb38e3ccbeff667-0-listening20200829T003553044Z.mp4'
2020-08-29T00:35:54.493146+00:00 app[web.1]:     }
2020-08-29T00:35:54.493146+00:00 app[web.1]:   },
2020-08-29T00:35:54.493147+00:00 app[web.1]:   userId: '5ebda7b58fb38e3ccbeff667',
2020-08-29T00:35:54.493148+00:00 app[web.1]:   _id: '5f49a2eabeb3fb0017b8c0f5'
2020-08-29T00:35:54.493149+00:00 app[web.1]: }
2020-08-29T00:35:54.493387+00:00 app[web.1]: Generating a social preview image for parentId:5f2b38317f20777395c58939
2020-08-29T00:35:54.496268+00:00 app[web.1]: generated social preview image
2020-08-29T00:35:54.554310+00:00 app[web.1]: site: https://undebate-djbowers.herokuapp.com/realtime-smpreview
2020-08-29T00:35:54.554362+00:00 app[web.1]: Generate preview image for site: https://undebate-djbowers.herokuapp.com/realtime-smpreview image_file_name: site_preview_2020-8-29.png
2020-08-29T00:35:54.625912+00:00 app[web.1]: disconnected from socket: ryib_nnCpOFI0GieAAAA
2020-08-29T00:35:54.584402+00:00 heroku[router]: at=info method=GET path="/socket.io/?EIO=3&transport=polling&t=NGtf6Fe&b64=1&sid=ryib_nnCpOFI0GieAAAA" host=undebate-smpreview.herokuapp.com request_id=fc3d7ffe-67d5-43f6-bdfd-451a21abc5aa fwd="3.88.33.78" dyno=web.1 connect=0ms service=86ms status=200 bytes=242 protocol=http
2020-08-29T00:35:54.632375+00:00 heroku[router]: at=info method=GET path="/socket.io/?EIO=3&transport=websocket&sid=ryib_nnCpOFI0GieAAAA" host=undebate-smpreview.herokuapp.com request_id=50a1d0ef-a4be-4953-88c0-06288292a8ae fwd="3.88.33.78" dyno=web.1 connect=0ms service=163ms status=101 bytes=175 protocol=http
2020-08-29T00:35:58.212183+00:00 app[web.1]: * File Upload to Cloudinary http://res.cloudinary.com/hrewc5ehd/image/upload/v1598661357/k0wlwdhaszkze62mhvas.png

undebate server / socket-api client

2020-08-29T00:35:52.068836+00:00 app[web.1]: [2020-08-29T00:35:52.068] [INFO] browser - 2020-08-29T00:35:52.019Z Undebate.onUserUpload { socketId: 'I-c4nNZ29k5X9ox9AAAC',
2020-08-29T00:35:52.068847+00:00 app[web.1]:   userId: '5ebda7b58fb38e3ccbeff667' }
2020-08-29T00:35:52.079597+00:00 app[web.1]: [2020-08-29T00:35:52.079] [INFO] browser - 2020-08-29T00:35:52.020Z Undebate.onUserUpload { socketId: 'I-c4nNZ29k5X9ox9AAAC',
2020-08-29T00:35:52.079599+00:00 app[web.1]:   userId: '5ebda7b58fb38e3ccbeff667' }
2020-08-29T00:35:54.442915+00:00 app[web.1]: connected to socket: ryib_nnCpOFI0GieAAAA
2020-08-29T00:35:54.471844+00:00 app[web.1]: socket api called with args: generate_smpreview Iota {
2020-08-29T00:35:54.471845+00:00 app[web.1]:   parentId: '5f2b38317f20777395c58939',
2020-08-29T00:35:54.471845+00:00 app[web.1]:   subject: 'Participant:realtime-smpreview-Candidate Recorder',
2020-08-29T00:35:54.471846+00:00 app[web.1]:   description:
2020-08-29T00:35:54.471847+00:00 app[web.1]:    'A participant in the following discussion:A Candidate Recorder for the Conversation: realtime-smpreview',
2020-08-29T00:35:54.471847+00:00 app[web.1]:   component:
2020-08-29T00:35:54.471848+00:00 app[web.1]:    { component: 'MergeParticipants',
2020-08-29T00:35:54.471848+00:00 app[web.1]:      participant:
2020-08-29T00:35:54.471848+00:00 app[web.1]:       { speaking: [Array],
2020-08-29T00:35:54.471849+00:00 app[web.1]:         name: 'Test Heroku 4',
2020-08-29T00:35:54.471849+00:00 app[web.1]:         listening:
2020-08-29T00:35:54.471851+00:00 app[web.1]:          'https://res.cloudinary.com/hrewc5ehd/video/upload/q_auto/v1598661353/5ebda7b58fb38e3ccbeff667-0-listening20200829T003553044Z.mp4' } },
2020-08-29T00:35:54.471852+00:00 app[web.1]:   userId: '5ebda7b58fb38e3ccbeff667',
2020-08-29T00:35:54.471852+00:00 app[web.1]:   _id: 5f49a2eabeb3fb0017b8c0f5 }
2020-08-29T00:35:54.471969+00:00 app[web.1]: disconnected from socket: ryib_nnCpOFI0GieAAAA
2020-08-29T00:35:56.720319+00:00 heroku[router]: at=info method=GET path="/socket.io/socket.io.js" host=undebate-djbowers.herokuapp.com request_id=cab3598f-d5f5-46ee-8813-30e1c1d2a5a8 fwd="52.207.245.146" dyno=web.1 connect=0ms service=5ms status=200 bytes=68890 protocol=https
2020-08-29T00:35:56.622419+00:00 app[web.1]: [2020-08-29T00:35:56.622] [INFO] node - GET /realtime-smpreview undebate social media bot { browserConfig:
2020-08-29T00:35:56.622438+00:00 app[web.1]:    '{"os":{"name":"Unknown","version":[],"versionString":"Unknown"},"browser":{"name":"Unknown","version":[],"versionString":"Unknown"},"type":"bot","model":"","ip":"52.207.245.146"}' }
2020-08-29T00:35:56.668858+00:00 app[web.1]: [2020-08-29T00:35:56.668] [INFO] node - index browser does not support ES6
2020-08-29T00:35:56.675106+00:00 heroku[router]: at=info method=GET path="/realtime-smpreview" host=undebate-djbowers.herokuapp.com request_id=f1cd9740-7ad4-43cb-ac40-6bc4928d0c58 fwd="52.207.245.146" dyno=web.1 connect=0ms service=56ms status=200 bytes=7235 protocol=https
2020-08-29T00:35:56.802088+00:00 heroku[router]: at=info method=GET path="/assets/webpack/main.js" host=undebate-djbowers.herokuapp.com request_id=96bc0dbe-4f0c-43c2-ac89-2b7fd764ae86 fwd="52.207.245.146" dyno=web.1 connect=0ms service=65ms status=200 bytes=287118 protocol=https
2020-08-29T00:35:56.767927+00:00 heroku[router]: at=info method=GET path="/assets/js/socket.io-stream.js" host=undebate-djbowers.herokuapp.com request_id=8087081d-70ec-41dc-85ae-a1b9e6f3ff72 fwd="52.207.245.146" dyno=web.1 connect=1ms service=20ms status=200 bytes=50190 protocol=https
2020-08-29T00:35:57.116558+00:00 heroku[router]: at=info method=GET path="/socket.io/?EIO=3&transport=polling&t=NGtf6uO" host=undebate-djbowers.herokuapp.com request_id=6b59e440-7d02-4da7-8149-047c1c5954af fwd="52.207.245.146" dyno=web.1 connect=0ms service=1ms status=200 bytes=339 protocol=https
2020-08-29T00:35:57.236896+00:00 heroku[router]: at=info method=GET path="/socket.io/?EIO=3&transport=polling&t=NGtf6wl&sid=k8abTuAU71pq48k6AAAD" host=undebate-djbowers.herokuapp.com request_id=1df716dd-5ff4-4fa5-8b3e-283c5e0482c6 fwd="52.207.245.146" dyno=web.1 connect=0ms service=1ms status=200 bytes=288 protocol=https
2020-08-29T00:35:57.264801+00:00 heroku[router]: at=info method=POST path="/socket.io/?EIO=3&transport=polling&t=NGtf6x4&sid=k8abTuAU71pq48k6AAAD" host=undebate-djbowers.herokuapp.com request_id=1d1daac9-0520-4202-a5b1-d5c49b991ba5 fwd="52.207.245.146" dyno=web.1 connect=0ms service=7ms status=200 bytes=303 protocol=https
2020-08-29T00:35:57.368664+00:00 heroku[router]: at=info method=GET path="/socket.io/?EIO=3&transport=polling&t=NGtf6xA&sid=k8abTuAU71pq48k6AAAD" host=undebate-djbowers.herokuapp.com request_id=7516f0e6-a5d2-4339-9a20-55a9bf9bf4d4 fwd="52.207.245.146" dyno=web.1 connect=1ms service=105ms status=200 bytes=242 protocol=https
2020-08-29T00:35:57.260022+00:00 app[web.1]: [2020-08-29T00:35:57.259] [INFO] browser - 2020-08-29T00:35:57.121Z client main running on browser { socketId: 'k8abTuAU71pq48k6AAAD', userId: 'anonymous' }
2020-08-29T00:35:57.728292+00:00 heroku[router]: at=info method=GET path="/socket.io/?EIO=3&transport=websocket&sid=k8abTuAU71pq48k6AAAD" host=undebate-djbowers.herokuapp.com request_id=ff079111-8679-49f7-873d-bfb7d528f16c fwd="52.207.245.146" dyno=web.1 connect=0ms service=474ms status=101 bytes=183 protocol=https

@ddfridley I will publish the PR's now and move the issue to needs testing. Let me know if you have any issues deploying to Heroku or testing the code out.

You will need the following config vars for each server

smpreview

undebate

I am open to changing the names of the ENV variables as well. We could keep them generic in the socket-api repo and then overwrite them with something more specific for their implementations? It might be better to do something like this, at least for the client implementations. If we have more than one socket-api in undebate, we will have to change the name anyway.

undebate

Also, you mentioned security and documentation for this issue. I'd like to have you test it out and ok the functionality before I write the docs, but I'll happily do that before we close this issue. And for security, we're just using the SOCKET_API_KEYS but I think that's secure enough for our purposes?

djbowers commented 4 years ago

There are actually 3 tasks remaining on this, but none of them prevent testing.

Example ERR_CONNECTION_REFUSED error

$ npm start

> smpreview@0.0.1 start /Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview
> node main.js

(node:17298) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
Iota.init count 141
got new records: 1
Found a need to create or update the social preview image for parentId:5f2b38317f20777395c58939
site: http://localhost:3011/realtime-smpreview
Generate preview image for site: http://localhost:3011/realtime-smpreview image_file_name: site_preview_2020-8-28.png
(node:17298) UnhandledPromiseRejectionWarning: Error: net::ERR_CONNECTION_REFUSED at http://localhost:3011/realtime-smpreview
    at navigate (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/node_modules/puppeteer/lib/FrameManager.js:120:37)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async FrameManager.navigateFrame (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/node_modules/puppeteer/lib/FrameManager.js:94:17)
    at async Frame.goto (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/node_modules/puppeteer/lib/FrameManager.js:406:12)
    at async Page.goto (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/node_modules/puppeteer/lib/Page.js:672:12)
    at async _generateSitePreview (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/generate-smpreview.js:111:3)
    at async generateSMPreviewForParent (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/generate-smpreview.js:49:3)
    at async performDbScan (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/generate-smpreview.js:35:7)
  -- ASYNC --
    at Frame.<anonymous> (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/node_modules/puppeteer/lib/helper.js:111:15)
    at Page.goto (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/node_modules/puppeteer/lib/Page.js:672:49)
    at Page.<anonymous> (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/node_modules/puppeteer/lib/helper.js:112:23)
    at _generateSitePreview (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/generate-smpreview.js:111:14)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async generateSMPreviewForParent (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/generate-smpreview.js:49:3)
    at async performDbScan (/Users/djbowers/Code/2-Areas/HackforLA/EnCiv/smpreview/generate-smpreview.js:35:7)
(node:17298) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:17298) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
djbowers commented 4 years ago

Docs are finished for socket-api.

Just again for reference, here are the 3 PR's that need to be reviewed.

Once the socket-api PR is merged, I will update the package.json for undebate and smpreview to point to Enciv and we can merge those two PR's as well.

I think handling the ERR_CONNECTION_REFUSED for the smpreview server could be a separate issue, might even be a good first issue for someone.

djbowers commented 4 years ago

@ddfridley I debugged the issue that came up last week when we were on that call with Dana.

So we know Heroku puts its dynos to sleep. What was happening was when the smpreview server would wake back up to start accepting requests from undebate over socket.io, it would run its init script and do a db scan. So it was waking up, scanning and generating the new preview, then accepting the incoming request to create the same preview again, causing some weird problems.

To fix this, I moved the manual scan to its own file and gave it its own npm script, npm run scan. This way we can still use the manual scan feature locally, but when we start the server on Heroku, it will just start listening right away and not run the full scan every time.

I tested this out on my Heroku setup and it generated the preview as expected. I think at this point it is ready for review! Let me know if you have any questions or need help testing.

ddfridley commented 4 years ago

DJ

I think it would be more robust if we always did the scan.  That will prevent things from getting missed.

But I understand the issue.

You said "causing some weird problems" - let's figure out more.  One of the things I had to do on the BP Info server is allow socket.io messages accepted and queued and responded to after everything was initialized. If we want too long to respond to the socket.io message - then it generates an error and a retry which may cause "weird" problems.

When starting up, we could make a list of participant records  that have trigger new preview images, and if the new request is in the list then ignore it.

Or -

Can we run the check process after processing incoming requests.

D.

⁣Get BlueMail for Android ​

On Sep 9, 2020, 10:57 AM, at 10:57 AM, DJ Bowers notifications@github.com wrote:

@ddfridley I debugged the issue that came up last week when we were on that call with Dana.

So we know Heroku puts its dynos to sleep. What was happening was when the smpreview server would wake back up to start accepting requests from undebate over socket.io, it would run its init script and do a db scan. So it was waking up, scanning and generating the new preview, then accepting the incoming request to create the same preview again, causing some weird problems.

To fix this, I moved the manual scan to its own file and gave it its own npm script, npm run scan. This way we can still use the manual scan feature locally, but when we start the server on Heroku, it will just start listening right away and not run the full scan every time.

I tested this out on my Heroku setup and it generated the preview as expected. I think at this point it is ready for review! Let me know if you have any questions or need help testing.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/EnCiv/undebate/issues/237#issuecomment-689723462

djbowers commented 4 years ago

@ddfridley ok I see your point. The error and retry makes sense with the messages I was seeing, I will do some more investigation.