Daily digest cron job - Githubissues

newcobrian commented 6 years ago

Ideally works like:

for every hour, send a daily digest to any user that its 10am in their timezone
content includes: their mentions, comments to any of their posts, or responses to their comments. Number of threads added to their channels with the titles and posters (this might be overkill)

here's my commit: https://github.com/newcobrian/collabo/commit/a735385fd2ff198a5e85d6b8fd507025e2be73fe

and here's the docs I was referencing:

@newcodave - if you can get the cron job to just run, I can probably fill in the content portion and email send of it

Here's the output from logging. Looks like there's an hourly_job function but it's not doing anything:

Brian-Ls-MacBook-Air:collabo airbrian$ firebase functions:log --project collabo-bc9b2

┌──────────────────────────────────────────────┐ │ Update available: 4.2.1 (current: 4.0.3) │ │ Run npm install -g firebase-tools to update. │ └──────────────────────────────────────────────┘

2018-09-06T00:03:05.981504225Z D watchFile: Function execution started 2018-09-06T00:03:07.307018465Z D watchFile: Function execution took 1326 ms, finished with status code: 200 2018-09-06T00:03:07.949688996Z D watchFile: Function execution started 2018-09-06T00:03:07.956765867Z D watchFile: Function execution took 8 ms, finished with status code: 200 2018-09-06T00:03:35.776Z E watchFile: Unhandled rejection 2018-09-06T00:03:37.075Z E watchFile: Error: File not found: 1OeaiH9f2n5aHpbANE_iE7fHTraTkWvtPFwHzEU7rY94. at createError (/user_code/node_modules/googleapis/node_modules/axios/lib/core/createError.js:16:15) at settle (/user_code/node_modules/googleapis/node_modules/axios/lib/core/settle.js:18:12) at Unzip.handleStreamEnd (/user_code/node_modules/googleapis/node_modules/axios/lib/adapters/http.js:201:11) at emitNone (events.js:91:20) at Unzip.emit (events.js:185:7) at endReadableNT (_stream_readable.js:974:12) at _combinedTickCallback (internal/process/next_tick.js:80:11) at process._tickDomainCallback (internal/process/next_tick.js:128:9) 2018-09-06T00:06:37.898287301Z D watchFile: Function execution started 2018-09-06T00:06:39.307413109Z D watchFile: Function execution took 1410 ms, finished with status code: 200 2018-09-06T00:06:53.752463115Z D watchFile: Function execution started 2018-09-06T00:06:53.759870163Z D watchFile: Function execution took 8 ms, finished with status code: 200 2018-09-06T00:07:37.083080231Z D watchFile: Function execution started 2018-09-06T00:07:38.002450776Z D watchFile: Function execution took 920 ms, finished with status code: 200 2018-09-06T00:07:38.201621156Z D watchFile: Function execution started 2018-09-06T00:07:38.209794470Z D watchFile: Function execution took 9 ms, finished with status code: 200 2018-09-06T00:08:00.252589049Z D watchFile: Function execution started 2018-09-06T00:08:00.411225776Z D watchFile: Function execution started 2018-09-06T00:08:01.486922047Z D watchFile: Function execution took 1235 ms, finished with status code: 200 2018-09-06T00:08:01.660287754Z D watchFile: Function execution took 1250 ms, finished with status code: 200 2018-09-06T00:11:00.776807351Z D watchFile: Function execution started 2018-09-06T00:11:02.109015350Z D watchFile: Function execution took 1333 ms, finished with status code: 200 2018-09-06T00:11:02.651126536Z D watchFile: Function execution started 2018-09-06T00:11:03.160577650Z D watchFile: Function execution started 2018-09-06T00:11:03.837132676Z D watchFile: Function execution took 1187 ms, finished with status code: 200 2018-09-06T00:11:04.373749978Z D watchFile: Function execution took 1214 ms, finished with status code: 200 2018-09-06T22:16:53.415146274Z D watchFile: Function execution started 2018-09-06T22:16:53.874095519Z D watchFile: Function execution started 2018-09-06T22:16:54.687339347Z D watchFile: Function execution took 1273 ms, finished with status code: 200 2018-09-06T22:16:55.120364221Z D watchFile: Function execution took 1247 ms, finished with status code: 200 2018-09-06T23:26:53.810Z I : undefined 2018-09-06T23:27:04.144Z N hourly_job: undefined 2018-09-06T23:27:04.162Z N watchFile: undefined 2018-09-06T23:27:04.352Z N verify: undefined 2018-09-06T23:27:16.955Z N hourly_job: undefined 2018-09-06T23:27:17.459Z N verify: undefined 2018-09-06T23:27:18.417Z N watchFile: undefined

newcodave commented 6 years ago

OK I see why this doesn't work. We have some historical stuff on App Engine because this was based on Views, and the views stuff is still there in app.yaml but not hooked up. This project is solely Firebase hosted and so there's no Python stuff.

It is actually extremely ghetto that Firebase delegates to App Engine for cron, but it is what it is

newcodave commented 6 years ago

sorry about the latency here... are we still using INBOX_SEND_EMAIL_URL to send mail @newcobrian ?

newcobrian commented 6 years ago

yeah still using that to send all emails. Tried to switch from myviews to joinkoi but realized it's probably not setup on the collabo App Engine account so I just left it for now

newcobrian commented 6 years ago

hey @newcodave, seemed like you knew what the issue was here. Can you point me in the right direction and I'll look at this?

newcodave commented 6 years ago

looking into this today @newcobrian

newcodave commented 6 years ago

2018-10-20T23:10:55.017986715Z D hourly_job: Function execution started
2018-10-20T23:10:56.467Z I hourly_job: This job is run every hour!
2018-10-20T23:10:56.474Z E hourly_job: Function returned undefined, expected Promise or value
2018-10-20T23:10:56.491620782Z D hourly_job: Function execution took 1475 ms, finished with status: 'ok'

newcodave commented 6 years ago

I can't get the front end to compile right now, getting

static/js/main.17931690.js from UglifyJs SyntaxError: Unexpected token: operator (>) [./~/quill-magic-url/dist/index.js:1,15305]

newcodave commented 6 years ago

I can't get the front end to compile right now, getting

static/js/main.17931690.js from UglifyJs SyntaxError: Unexpected token: operator (>) [./~/quill-magic-url/dist/index.js:1,15305]

... but I merged this down to master. Cheers

newcobrian commented 6 years ago

niiice thanks. Looks like I can just write some js in here to run through firebase and send emails?

app.get('/publish/:topic', async(req, res) => {

newcodave commented 6 years ago

You want to put your code here, in functions/index.js:

console.log("This job is run every hour!")

You can trigger the queue publish, and the function itself, from the command line, by hitting the endpoint that App Engine cron hits every hour:

% curl http://collabo-bc9b2.appspot.com/publish/hourly-tick

This wasn't done when I thought it was done. But it should be done now :) I'll double-check and make sure the cron is firing correctly. There's also a panel in Google Cloud Console, [Side Menu] > App Engine > Cron jobs where you can just press Run Now

newcodave commented 6 years ago

Oh, and I also modified package.json at the top level so that npm run deploy deploys all the functions. If you're only working on the firebase functions though, you can do firebase deploy --functions, I think, to update just functions/index.js

newcobrian commented 6 years ago

ok thanks

newcodave commented 6 years ago

Just checked this again, looks like it's firing hourly. Gonna go ahead and make

2018-10-21T18:23:01.954Z E hourly_job: Function returned undefined, expected Promise or value 2018-10-21T18:23:02.011028158Z D hourly_job: Function execution took 79 ms, finished with status: 'ok' 2018-10-21T19:23:03.039706403Z D hourly_job: Function execution started 2018-10-21T19:23:04.767Z I hourly_job: This job is run every hour! 2018-10-21T19:23:04.776Z E hourly_job: Function returned undefined, expected Promise or value 2018-10-21T19:23:04.794886583Z D hourly_job: Function execution took 1756 ms, finished with status: 'ok' 2018-10-21T20:23:03.326849586Z D hourly_job: Function execution started 2018-10-21T20:23:05.166Z I hourly_job: This job is run every hour! 2018-10-21T20:23:05.177Z E hourly_job: Function returned undefined, expected Promise or value 2018-10-21T20:23:05.200637042Z D hourly_job: Function execution took 1875 ms, finished with status: 'ok' 2018-10-21T21:23:03.734211950Z D hourly_job: Function execution started 2018-10-21T21:23:05.329Z I hourly_job: This job is run every hour! 2018-10-21T21:23:05.338Z E hourly_job: Function returned undefined, expected Promise or value 2018-10-21T21:23:05.359045565Z D hourly_job: Function execution took 1626 ms, finished with status: 'ok' 2018-10-21T22:23:03.509010106Z D hourly_job: Function execution started 2018-10-21T22:23:05.212Z I hourly_job: This job is run every hour! 2018-10-21T22:23:05.221Z E hourly_job: Function returned undefined, expected Promise or value 2018-10-21T22:23:05.239263076Z D hourly_job: Function execution took 1731 ms, finished with status: 'ok' 2018-10-21T23:23:03.894804993Z D hourly_job: Function execution started 2018-10-21T23:23:05.703Z I hourly_job: This job is run every hour! 2018-10-21T23:23:05.711Z E hourly_job: Function returned undefined, expected Promise or value 2018-10-21T23:23:05.730879344Z D hourly_job: Function execution took 1837 ms, finished with status: 'ok' 2018-10-22T00:23:04.353942163Z D hourly_job: Function execution started 2018-10-22T00:23:06.085Z I hourly_job: This job is run every hour! 2018-10-22T00:23:06.097Z E hourly_job: Function returned undefined, expected Promise or value 2018-10-22T00:23:06.113480510Z D hourly_job: Function execution took 1761 ms, finished with status: 'ok' 2018-10-22T01:23:03.445770651Z D hourly_job: Function execution started 2018-10-22T01:23:05.250Z I hourly_job: This job is run every hour! 2018-10-22T01:23:05.259Z E hourly_job: Function returned undefined, expected Promise or value 2018-10-22T01:23:05.279427710Z D hourly_job: Function execution took 1834 ms, finished with status: 'ok'

newcodave commented 6 years ago

... the daily and weekly functions also:

iMac [~/D/c/appengine:master]> firebase functions:log -n 1000|grep dail (node:33636) ExperimentalWarning: The fs.promises API is experimental 2018-10-22T01:53:05.547Z N daily_job: undefined 2018-10-22T01:53:42.946Z N daily_job: undefined 2018-10-22T01:54:22.614051078Z D daily_job: Function execution started 2018-10-22T01:54:22.876Z I daily_job: This job is run every day! 2018-10-22T01:54:22.886Z E daily_job: Function returned undefined, expected Promise or value 2018-10-22T01:54:22.979099866Z D daily_job: Function execution took 366 ms, finished with status: 'ok' iMac [~/D/c/appengine:master]> firebase functions:log -n 1000|grep week (node:33800) ExperimentalWarning: The fs.promises API is experimental 2018-10-22T01:53:05.537Z N weekly_job: undefined 2018-10-22T01:53:40.874Z N weekly_job: undefined 2018-10-22T01:55:11.476673256Z D weekly_job: Function execution started 2018-10-22T01:55:11.831Z I weekly_job: This job is run every week! 2018-10-22T01:55:11.839Z E weekly_job: Function returned undefined, expected Promise or value 2018-10-22T01:55:11.928209906Z D weekly_job: Function execution took 452 ms, finished with status: 'ok'

newcobrian commented 6 years ago

ok thanks

On Sat, Oct 20, 2018 at 7:27 PM David Young notifications@github.com wrote:

Oh, and I also modified package.json at the top level so that npm run deploy deploys all the functions. If you're only working on the firebase functions though, you can do firebase deploy --functions, I think, to update just functions/index.js

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/newcobrian/collabo/issues/20#issuecomment-431633612, or mute the thread https://github.com/notifications/unsubscribe-auth/AKorPhmk8C_l337sF5jAZRne21Cir3jKks5um9uDgaJpZM4Wfoxv .

newcobrian commented 5 years ago

is there a faster way to test than deploying then hitting:

% curl http://collabo-bc9b2.appspot.com/publish/hourly-tick

newcodave commented 5 years ago

Locally you could use the node command line or the CLI to call the function, does that make sense? But it won’t be a super similar execution environment.

I think this is a reasonable guide to wrapping a function in a CLI, but haven’t done this myself

https://www.sitepoint.com/javascript-command-line-interface-cli-node-js/

—dave On Nov 28, 2018, 12:00 -0800, Brian notifications@github.com, wrote:

is there a faster way to test than deploying then hitting: % curl http://collabo-bc9b2.appspot.com/publish/hourly-tick — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

newcobrian commented 5 years ago

think I got the daily digest email to work on master. Currently only sending to my user Id for the collabo org. Sometimes when testing, I get the send timeout below. Also have seen issues where some people aren't getting emails when they should for our normal notification emails. Is it possible that Campaign Monitor just drops emails? Is there a good way to track this?

2018-12-03T23:01:18.424Z I hourly_job: { FetchError: request to https://myviews.io/mail/send failed, reason: connect ETIMEDOUT 192.64.119.119:443 at ClientRequest. (/user_code/node_modules/node-fetch/lib/index.js:1444:11) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at TLSSocket.socketErrorListener (_http_client.js:310:9) at emitOne (events.js:96:13) at TLSSocket.emit (events.js:188:7) at emitErrorNT (net.js:1296:8) at _combinedTickCallback (internal/process/next_tick.js:80:11) at process._tickDomainCallback (internal/process/next_tick.js:128:9) message: 'request to https://myviews.io/mail/send failed, reason: connect ETIMEDOUT 192.64.119.119:443', type: 'system', errno: 'ETIMEDOUT', code: 'ETIMEDOUT' }

newcobrian commented 5 years ago

hm turning this on for everyone now since I think it's working.

Couple follow on questions: 1) would it be too many db reads per day to fetch everyone's inbox during the cron job and include notifs? 2) Is there a good way to make sure Campaign Monitor is sending the emails?

newcodave commented 5 years ago

re 1, probably not until we got much larger - this is a pretty standard amount of stuff to look at for e-mails

re 2, campaign monitor has a dashboard at

https://futurehumansllc.createsend.com/triggered/campaigns/2BCC71CE56E54A89

... is a high level dashboard, and

https://futurehumansllc.createsend.com/triggered/reports/2bcc71ce56e54a89#/?tag=all&timeframe=all

... lets you see every sent message by its delivery status

newcobrian / collabo

Daily digest cron job #20