S+H: left menu messed up again

[KathJohnson]

OK, there is a problem with the left hand menu for S+H again. This time I am completely 2000% certain I have not deployed the code to that site, so it may be a good time to suss out how this is happening. I did deploy code to another site on the same server today (historichouseguy). I cannot confirm that's when it happened, but it may be related, so I am including it as possibly helpful information.

@1sp and @jboz62, can one of your restore the backup from last week for this site (shconstruction.com)?

@1sp, you are the Forge expert, I think. Here is a screenshot of the deploy code for that site, in case that is helpful. Please let me know what I can do to help you figure out what happened. I have not changed any of the deploy code, even though that is under discussion.

[1sp]

@KathJohnson The deploy script in the screenshot is for shconstruction2.corbettresearchgroup.com .

Assuming that you deployed for shconstruction2 , the problem after deployment is a known one as I didn't started on the fix for issue we uncovered during the similar incident on S+H site in past (Please see https://github.com/meetinghouse/cms/issues/252#issuecomment-220109326)

[1sp]

@KathJohnson Can you please share the forge ssh access via email?

[meetinghouse]

shconstruction2 is the staging site for shconstruction.com. When we launch a site live, it basically is a copy of the staging site. The staging site stays on the server and code is deployed there. I hope that makes sense.

I did not deploy code to shconstruction2.

I understand the problem hasn't been corrected yet. While we hammer out a safer process, I was thinking the shconstruction site could site safely, without having any code deployments to it, until we're ready to roll out the fix. But somehow something happened. There must be something somewhere that is making it happen, but I can't see what it might be.

[KathJohnson]

I've sent you the credentials I use to login to Forge via email. Not sure that's what you need, though.

[1sp]

@meetinghouse I am so sorry for the confusion - I took the site in the screenshot as site what was actually deployed (in context to "Here is a screenshot of the deploy code for that site, in case that is helpful.")

@KathJohnson While we discuss this out, can you please share ssh/sftp access to the forge so that I may rollback the site's database to pre-deployment state quickly ? Thank you!.

[edited to correct the mention]

[1sp]

@KathJohnson I am yet to see your email in my inbox - I am sure it is lurking somewhere around :)

Last time rollback was done using @jboz62 's system . I think he has an email/note with login credentials which I used. Please send me the same information.

[KathJohnson]

@1sp Is what I just sent you a few minutes ago going to work? This is what I sent to John B. on May 10.

I also have some RSync credentials if that is what you need.

[1sp]

@KathJohnson Sorry, I do not have any new email from you (just for confirmation my email is edited by @1sp).

To start work I would need forge's ssh access. I already have the rsync.net credentials and instructions in my email and Google drive.

[1sp]

@KathJohnson I got the email. Thank you!

[1sp]

@KathJohnson No, this access won't work. I would need a command-line shell or SFTP access so that I may directly access and modify the files on the server.

Do you have a email from digital ocean with mention of SSH access? I am sure @jboz62 can give you more information.

[jboz62]

Sudhir, let me see if I can find that.

On May 24, 2016, at 2:45 PM, Sudhir notifications@github.com wrote:

@KathJohnson https://github.com/KathJohnson Sorry, I do not have any new email from you (just for confirmation my email is edited by @1sp https://github.com/1sp).

To start work I would need forge's ssh access. I already have the rsync.net credentials and instructions in my email and Google drive.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/meetinghouse/cms/issues/260#issuecomment-221360931

[1sp]

@jboz62 thanks John

[jboz62]

Sudhir, I sent creds to your gmail account, please check that.

On May 24, 2016, at 3:05 PM, Sudhir notifications@github.com wrote:

@jboz62 https://github.com/jboz62 thanks John

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/meetinghouse/cms/issues/260#issuecomment-221367851

[jboz62]

Were you able to get in, Sudhir?

On May 24, 2016, at 3:05 PM, Sudhir notifications@github.com wrote:

@jboz62 https://github.com/jboz62 thanks John

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/meetinghouse/cms/issues/260#issuecomment-221367851

[1sp]

Thanks John, the credentials worked!

I have rolled back the site to its old state. Please review.

[jboz62]

Yes, I see that, thank you! Any ideas on how this keeps happening?

On May 24, 2016, at 4:15 PM, Sudhir notifications@github.com wrote:

Thanks John, the credentials worked!

I have rolled back the site to its old state. Please review.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/meetinghouse/cms/issues/260#issuecomment-221380118

[1sp]

I reviewed the deployment script and found nothing alarming. The deployment log also makes sense.

It could be that we have uncovered a bug in forge. I would suggest to put a command auditor/logger on the server for some time to see the process execution flow during the forge deployment.

[meetinghouse]

Thank you so much for rolling it back and looking at the code for problems.

You said way back at the beginning that hardly anyone uses Forge for deployment. From your perspective, would it be a good idea to abandon Forge for code updates and do them directly in Digital Ocean instead? Is that a fairly user-friendly environment a non-programmer could manage?

The goal is to have one code base that can (ultimately) deploy automatically across all sites so we have one product to manage. We're willing to compromise on that goal a bit to avoid this damaging unpredictability, though.

@acavanaugh, I want to call your attention to this question in particular, since you may have some insight as to why we are using Forge for ongoing code deployment. Thanks.

Sincerely, Katherine Johnson, Research Director Corbett Research Group http://www.restorationtrades.com "Web Marketing that Works"

Publishers of:

The Custom Building & Restoration Trades Directory http://www.restorationtradesdirectory.com

The Building Arts Notebook, "Journal of the Guild of Building Artisans" http://buildingartisansguild.com/

Email: katherine@restorationtrades.com Phone: (413) 475-3154

On Tue, May 24, 2016 at 6:34 PM, Sudhir notifications@github.com wrote:

I reviewed the deployment script and found nothing alarming. The deployment log also makes sense.

It could be that we have uncovered a bug in forge. I would suggest to put a command auditor/logger on the server for some time to see the process execution flow during the forge deployment.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/meetinghouse/cms/issues/260#issuecomment-221422365

[1sp]

Yes, whatever Forge is doing is trivial for moderately experienced systems guy via custom scripts. Since I see that we usually deploy the changes incrementally on specific sites - It would be a good idea to have a custom built console to do transactional deployments.

"The goal is to have one code base that can (ultimately) deploy automatically across all sites" - Though it looks like a fairly long term plan but we can also think about multi-site setup (the way wordpress and other cmses do it)

[acavanaugh]

@meetinghouse I believe you tagged me in error.

[meetinghouse]

Andy,

I actually did intend to tag you here. The reason is that you may know why it was (or maybe still is) important to use Forge for deployment of code from GitHub to the sites.

The reason I can think of is to enable us to auto-deploy changes across the board. We aren't doing that at the moment, but it is still a hope for the future. Am I correct in that, or is there some other reason you can think of why we are using Forge this way?

[acavanagh]

@meetinghouse that's the wrong acavanagh. I'm acavanagh with no 'u'. You tagged acavanaugh.

[acavanagh]

@1sp agreed that scripting this out would be better/easier for a moderately experienced systems guy - Laravel also ships with Envoy, which makes that even easier. However, they're looking for something with a web GUI interface - Literal push-button deployments, no command line, ssh etc. Hence forge.

[KathJohnson]

The deployment issue appears to have been solved, but we are keeping an eye on recent server events for any unexpected deployments.