search

jsha / blocktogether

Share your blocks and subscribe to others'
GNU General Public License v3.0
330 stars 68 forks source link

How to run Block Together in prod env #316

Closed lbarbaglia closed 1 year ago

lbarbaglia commented 4 years ago

Hi, You provide the steps to run blocktogether in a dev environment but can you also provide the steps for a production environment please ? So we can run our own blocktogether instance.

Thanks

vissjb commented 4 years ago

This would also be helpful to me. I anticipate standing up a small instance for my list subscribers.

jdbillingham commented 4 years ago

I've set my own one up at https://theblockbot.com, and blogged some rough instructions here - https://www.oolon.co.uk/?p=9

Not exactly "production ready", and running on my own home server. Made a couple of small modifications to it to disallow people sharing their blocklists without a username/password authentication. (Which is probably not very secure)

vissjb commented 4 years ago

I'm looking through your post and it's very helpful. I was thinking about just running it in "Development" mode in vagrant like you did, but I'd really like to get it running on VM. Right now I'm trying a google cloud instance VM and the capistrano deployment. Pretty finicky though.

On Sun, Jul 12, 2020 at 3:19 AM jdbillingham notifications@github.com wrote:

I've set my own one up at https://theblockbot.com, and blogged some rough instructions here - https://www.oolon.co.uk/?p=9

Not exactly "production ready", and running on my own home server. Made a couple of small modifications to it to disallow people sharing their blocklists without a username/password authentication. (Which is probably not very secure)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jsha/blocktogether/issues/316#issuecomment-657220894, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7TAIVA2NVUFIWFRR4ZBRLR3GZ65ANCNFSM4OJXWENA .

jdbillingham commented 4 years ago

I didn't run in vagrant, it was a local server, also changed from running in "development" mode to production with the latest change to make the processes run as services from systemd. But the only difference between "development" and "production" modes seem to be sequelize configuration is different and in dev mode it doesn't cache the mustache templates - flushes each time so you can see changes.

vissjb commented 4 years ago

So are you still running nohup ./run-dev.sh?

On Wed, Jul 15, 2020 at 11:44 PM jdbillingham notifications@github.com wrote:

I didn't run in vagrant, it was a local server, also changed from running in "development" mode to production with the latest change to make the processes run as services from systemd. But the only difference between "development" and "production" modes seem to be sequelize configuration is different and in dev mode it doesn't cache the mustache templates - flushes each time so you can see changes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jsha/blocktogether/issues/316#issuecomment-659301176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7TAIUKRJHCLCMQCFBZX3TR33DYHANCNFSM4OJXWENA .

vissjb commented 3 years ago

I just added some folks to my block list, but it's been four days since the list has updated. Any thoughts on how to troubleshoot that?

Thanks, John

On Thu, Jul 16, 2020 at 4:19 AM John Visser johnvisser@gmail.com wrote:

So are you still running nohup ./run-dev.sh?

On Wed, Jul 15, 2020 at 11:44 PM jdbillingham notifications@github.com wrote:

I didn't run in vagrant, it was a local server, also changed from running in "development" mode to production with the latest change to make the processes run as services from systemd. But the only difference between "development" and "production" modes seem to be sequelize configuration is different and in dev mode it doesn't cache the mustache templates - flushes each time so you can see changes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jsha/blocktogether/issues/316#issuecomment-659301176, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7TAIUKRJHCLCMQCFBZX3TR33DYHANCNFSM4OJXWENA .

jdbillingham commented 3 years ago

So are you still running nohup ./run-dev.sh? … On Wed, Jul 15, 2020 at 11:44 PM jdbillingham @.***> wrote: I didn't run in vagrant, it was a local server, also changed from running in "development" mode to production with the latest change to make the processes run as services from systemd. But the only difference between "development" and "production" modes seem to be sequelize configuration is different and in dev mode it doesn't cache the mustache templates - flushes each time so you can see changes. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#316 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7TAIUKRJHCLCMQCFBZX3TR33DYHANCNFSM4OJXWENA .

See the updates to the end of the blog post, I copied Jacobs systemd config, with a couple of changes. Now the processes run as a service.

jdbillingham commented 3 years ago

I just added some folks to my block list, but it's been four days since the list has updated. Any thoughts on how to troubleshoot that? Thanks, John … On Thu, Jul 16, 2020 at 4:19 AM John Visser @.> wrote: So are you still running nohup ./run-dev.sh? On Wed, Jul 15, 2020 at 11:44 PM jdbillingham @.> wrote: > I didn't run in vagrant, it was a local server, also changed from running > in "development" mode to production with the latest change to make the > processes run as services from systemd. But the only difference between > "development" and "production" modes seem to be sequelize configuration is > different and in dev mode it doesn't cache the mustache templates - flushes > each time so you can see changes. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#316 (comment)>, > or unsubscribe > https://github.com/notifications/unsubscribe-auth/AH7TAIUKRJHCLCMQCFBZX3TR33DYHANCNFSM4OJXWENA > . >

On your own instance of blocktogether? The update-blocks.js seems to be the process that pulls in the latest block list for people, updates it every interval. I did have a problem where a kill to the process made it sit and wait and not actually exit. Take a look at the end of the code, a SIGINT or SIGTERM results in it trying to shut down gracefully, but it waits for a second signal before actually exiting. I had 'Closing up shop.' in the logs and the process was doing nothing ... Until I killed it and it restarted properly.

knsankar commented 3 years ago

@jdbillingham Is settings page is protected in your instance? There is a basic auth when I try to access settings.

jdbillingham commented 3 years ago

@jdbillingham Is settings page is protected in your instance? There is a basic auth when I try to access settings.

Yes, I added an nginx config to protect the page so not anyone can share their list. Jacob had a 1Tb DB and a big server running blocktogether.org, I've got a small home server it runs on. So I can only support a small number of lists.

knsankar commented 3 years ago

@jdbillingham I'm facing a weird issue, the block is updating only if the owner of blocklist logins into the portal checks /actions.

update-blocks, update-users, blocktogether, actions, deleter services are running without any error. what could be the issue?

vissjb commented 3 years ago

They only update once every 24 hours or so, unless you log-in, in which case they update immediately. Have you gone a full 24 hours and not seen any updates?

On Sun, Aug 23, 2020 at 4:10 PM Sankar notifications@github.com wrote:

@jdbillingham https://github.com/jdbillingham I'm facing a weird issue, the block is updating only if the owner of blocklist logins into the portal checks /actions.

update-blocks, update-users, blocktogether, actions, deleter services are running without any error. what could be the issue?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jsha/blocktogether/issues/316#issuecomment-678831053, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7TAISIBXDIGOKHCNJIEPLSCGHTXANCNFSM4OJXWENA .

knsankar commented 3 years ago

@vissjb I checked after few hours. Let me wait for a day and check again.

Thank you :)

Update: he blocks were updated after few hours.

aana5i commented 3 years ago

Hello @jdbillingham i saw your try, nice one ! i have a question. i try to do the same as you but on the block page i only got the uid, not name, followers ect ... Seem's like they are not stored in DB. Are they catch on the fly ? Did you have an advice ?

jdbillingham commented 3 years ago

Hello @jdbillingham i saw your try, nice one ! i have a question. i try to do the same as you but on the block page i only got the uid, not name, followers ect ... Seem's like they are not stored in DB. Are they catch on the fly ? Did you have an advice ?

If you look right at the bottom of my post, Updates - 4. This is a known issue, one I've not found a fix for, but you can manually work around it by pushing the IDs into the table to force an update. It then seems to manage to update incremental changes OK. https://www.oolon.co.uk

aana5i commented 3 years ago

thanks @jdbillingham the IDs are in the DB. i just can't get the screen_name or other information about the blocked user's.

jdbillingham commented 3 years ago

thanks @jdbillingham the IDs are in the DB. i just can't get the screen_name or other information about the blocked user's.

Yes if you look at my blog post you need to push the IDs into the TwitterUsers table, the code then updates all those to have screen name etc. Insert into TwitterUsers (uid,createdAt,updatedAt) Select distinct sink_uid,now(),now() from Blocks Where BlockBatchId = “xxx” and sink_uid not in (select uid from TwitterUsers);

aana5i commented 3 years ago

i see, thanks. wil ltry that way.

Worked, thanks.

aana5i commented 3 years ago

Found @jdbillingham its not realy a bug but ...

on update-blocks.js row 240: if (user.shared_blocks_key != null && size != 125000) { user.shared_blocks_key is null if the user don't share he's block list. you need to go in setting page then hit 'Share your block list with friends'.

thansk again for your help