search

pythonindia / magudi

Salt stack based config for Python India servers
https://in.pycon.org
5 stars 9 forks source link

junction services failing in server #21

Closed ananyo2012 closed 5 years ago

ananyo2012 commented 6 years ago

So I did a systemctl restart junction to restart the junction service logged in as root. But then the junction services started failing in subsequent runs of provision script

Error Logs

----------
          ID: junction_code
    Function: git.latest
        Name: https://github.com/pythonindia/junction.git
      Result: False
     Comment: fatal: unable to read tree 5823dcb83dffbe122ca81ce20d52f4688627ad79
     Started: 10:52:35.894491
    Duration: 4488.691 ms
     Changes:
----------
          ID: /opt/envs/junction
    Function: virtualenv.managed
      Result: False
     Comment: One or more requisite failed: junction.junction_code
     Started: 10:52:40.385927
    Duration: 0.014 ms
     Changes:
----------
          ID: junction_uwsgi
    Function: pip.installed
        Name: uwsgi
      Result: False
     Comment: One or more requisite failed: junction./opt/envs/junction
     Started: 10:52:40.386622
    Duration: 0.013 ms
     Changes:
----------
          ID: junction
    Function: service.running
      Result: False
     Comment: One or more requisite failed: junction.junction_code, junction.junction_uwsgi
     Started: 10:52:40.457517
    Duration: 0.015 ms
     Changes:
----------
          ID: junction_pip_upgrade
    Function: pip.installed
        Name: pip
      Result: False
     Comment: One or more requisite failed: junction./opt/envs/junction
     Started: 10:52:40.561347
    Duration: 0.016 ms
     Changes:
----------
          ID: junction_pip_requirements
    Function: pip.installed
      Result: False
     Comment: One or more requisite failed: junction.junction_pip_upgrade
     Started: 10:52:40.562010
    Duration: 0.013 ms
     Changes:
----------
          ID: migrate_junction
    Function: cmd.run
        Name: /opt/envs/junction/bin/python manage.py migrate
      Result: False
     Comment: One or more requisite failed: junction.junction_pip_requirements
     Started: 10:52:41.730463
    Duration: 0.015 ms
     Changes:
----------
          ID: junction_celery
    Function: service.running
      Result: False
     Comment: One or more requisite failed: junction.junction_code, junction.migrate_junction
     Started: 10:52:41.733723
    Duration: 0.015 ms
     Changes:
----------
          ID: collectstatic_junction
    Function: cmd.run
        Name: /opt/envs/junction/bin/python manage.py collectstatic --noinput
      Result: False
     Comment: One or more requisite failed: junction./opt/envs/junction
     Started: 10:52:41.735046
    Duration: 0.013 ms
     Changes:
----------
          ID: /opt/junction/qr_codes/
    Function: file.directory
      Result: False
     Comment: One or more requisite failed: junction.junction_code
     Started: 10:52:41.736240
    Duration: 0.013 ms
     Changes:
ananyo2012 commented 6 years ago

@vigneshsarma Can you have a look ?

vigneshsarma commented 6 years ago

It seems like our clone of Junction is corrupted. Refer: https://stackoverflow.com/questions/20870895/how-to-diagnose-and-fix-git-fatal-unable-to-read-tree

This is what I am seeing, 

junction$ git reset --hard 5823dcb83dffbe122ca81ce20d52f4688627ad79
fatal: unable to read tree 5823dcb83dffbe122ca81ce20d52f4688627ad79

Sadly our git version is old and we don't have the command to fix the clone where it is. I cloned it to a tmp and retried it like so,

/tmp$ git clone https://github.com/pythonindia/junction.git
Cloning into 'junction'...
remote: Counting objects: 7240, done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 7240 (delta 19), reused 20 (delta 13), pack-reused 7196
Receiving objects: 100% (7240/7240), 3.92 MiB | 911.00 KiB/s, done.
Resolving deltas: 100% (5010/5010), done.
Checking connectivity... done.
root@in:/tmp$ cd junction/
/tmp/junction$ git reset --hard 5823dcb83dffbe122ca81ce20d52f4688627ad79
HEAD is now at 5823dcb Adding a Proposals API (#566)

So it seems like we have to delete the existing clone/folder and rerun provision. We should do this when the traffic is lowest. Also, we should make sure if there are info inside these folders that we might need like, uploads etc are preserved.

Can you help me with figuring out,

  1. what would be a safe time to take the site down and delete the folder?
  2. What all if any uncommitted info within the folder needs to be backed up?
ananyo2012 commented 6 years ago

It seems like our clone of Junction is corrupted. Refer: https://stackoverflow.com/questions/20870895/how-to-diagnose-and-fix-git-fatal-unable-to-read-tree

This is what I am seeing,

junction$ git reset --hard 5823dcb83dffbe122ca81ce20d52f4688627ad79 fatal: unable to read tree 5823dcb83dffbe122ca81ce20d52f4688627ad79

I think this is because the commit is not present in the current version. Can you check once? Also is git reset needed for pulling in a new change?

Sadly our git version is old and we don't have the command to fix the clone where it is.

Yes I do see git related errors when I try to run provision to pull in a new change for the PyCon India website. I will share the log when I get it next time

Can you help me with figuring out,

  1. what would be a safe time to take the site down and delete the folder?

I am not aware if we are tracking the website traffic. I will try to check with some folks if there is something already present. If its not there I will need to few days to check the traffic in case ww have to set it up.

  1. What all if any uncommitted info within the folder needs to be backed up?

Yeah then probably we have to reapply those Changes. I can do that. Let me first check the first point to begin with.

ananyo2012 commented 6 years ago

I think this is because the commit is not present in the current version. Can you check once? Also is git reset needed for pulling in a new change?

Ah ok git fetches the entire commit history with a git fetch or git ls-remote in this case I think. So salt uses git reset --hard <commit_hash> to get the latest commit. The link here https://stackoverflow.com/questions/20870895/how-to-diagnose-and-fix-git-fatal-unable-to-read-tree mentions that re-cloning might not solve the problem

ananyo2012 commented 6 years ago

I got the issue temporarily fixed by following these steps

git pull origin master
provision
systemctl restart junction

But we have to do this each time a new change is to be merged in junction

ananyo2012 commented 6 years ago

Also here is the issue I am facing while updating the inpycon2018 website. This occurs when I am running provision for the first time after update. But when I run provision for the second time it works fine

[INFO    ] Executing command [u'git', u'diff', u'HEAD'] in directory '/opt/inpycon2018'
[INFO    ] Executing command [u'git', u'rev-parse', u'481141cdb56e8c15743789b748414d92499b0ab4^{commit}'] in directory '/opt/inpycon2018'
[DEBUG   ] stdout: 481141cdb56e8c15743789b748414d92499b0ab4^{commit}
[DEBUG   ] stderr: fatal: ambiguous argument '481141cdb56e8c15743789b748414d92499b0ab4^{commit}': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
[DEBUG   ] retcode: 128
[INFO    ] Executing command [u'git', u'rev-parse', u'--abbrev-ref', u'master@{upstream}'] in directory '/opt/inpycon2018'
[DEBUG   ] stdout: origin/master
[INFO    ] Executing command [u'git', u'ls-remote', u'--tags', u'origin'] in directory '/opt/inpycon2018'
[INFO    ] Executing command [u'git', u'fetch', u'origin', u'refs/heads/*:refs/remotes/origin/*', u'+refs/tags/*:refs/tags/*'] in directory '/opt/inpycon2018'
[ERROR   ] Command '[u'git', u'fetch', u'origin', u'refs/heads/*:refs/remotes/origin/*', u'+refs/tags/*:refs/tags/*']' failed with return code: 1
[ERROR   ] stdout: From https://github.com/pythonindia/inpycon2018
 ! [rejected]        gh-pages   -> origin/gh-pages  (non-fast-forward)
   c842991..481141c  master     -> origin/master
[ERROR   ] retcode: 1
[ERROR   ] Fetch failed. Set 'force_fetch' to True to force the fetch if the failure was due to not being able to fast-forward. Output of the fetch command follows:

From https://github.com/pythonindia/inpycon2018
 ! [rejected]        gh-pages   -> origin/gh-pages  (non-fast-forward)
   c842991..481141c  master     -> origin/master
[INFO    ] Completed state [https://github.com/pythonindia/inpycon2018.git] at time 23:02:19.149350 (duration_in_ms=5402.166)

----------
          ID: https://github.com/pythonindia/inpycon2018.git
    Function: git.latest
      Result: False
     Comment: Fetch failed. Set 'force_fetch' to True to force the fetch if the failure was due to not being able to fast-forward. Output of the fetch command follows:

              From https://github.com/pythonindia/inpycon2018
               ! [rejected]        gh-pages   -> origin/gh-pages  (non-fast-forward)
                 c842991..481141c  master     -> origin/master
     Started: 23:02:13.747184
    Duration: 5402.166 ms
     Changes:
vigneshsarma commented 5 years ago

not happening any more