amilworks commented 4 years ago

TurboGears 2.3.0b2

This is the second upgrade in the process. If you have any questions, tag @amilworks on here. Current Version: TurboGears2==2.2.0 Upgrade Version: TurboGears2==2.3.0b2

As you are going through, check off what Step you completed to make it easier for us to help, and for you to feel accomplished.

Getting Started

Step 1. Clone the Repo

GOAL

[x] Start by cloning the branch from this current repo

git clone -b tg-upgrade --single-branch https://github.com/UCSB-VRL/bisqueUCSB.git

cd tg-upgrade and you should see a folder structure like this

.
├── boot
├── builder
├── COPYRIGHT
├── DOCKER-BUILD_LOG.md
├── Dockerfile.caffe.xenial
├── DOCKER-RUN_LOG.md
├── entry.sh
├── LICENSE
├── Makefile
├── modules
├── README.md
├── requirements.txt
├── run-bisque.sh
├── source
├── sources.list
├── start-bisque.sh
└── virtualenv.sh

4 directories, 13 files

Step 2. Verify Everything Works

GOAL

[x] Establish a baseline that BisQue works on your local system before changing anything.

You have two choices to verify that the repo is working.

Option 1. Pull the Latest Docker Build (Recommended)

The repo is rebuilt and pushed automatically after a successful push/merge on GitHub. Download the latest image by running the following command:

docker pull amilworks/bisque-tg2-upgrade:git

You can run this image to verify BisQue is working and able to run locally on your system by running:

docker run -itp 8080:8080  --name bqdev-tg2 amilworks/bisque-tg2-upgrade:git

You should see BisQue on http://localhost:8080/ in your browser. If not, paste the logs below and we will take a look.

Option 2. Build the Image

Building the image from scratch is what you will end up doing throughout the upgrade process. Luckily, we automated it to where you run a Makefile.

make bisque05-caffe-xenial

If you vim Makefile, you will see it's a series of basic commands. I suggest you Edit this file and --tag the image something different and update the version you are upgrading instead of amilworks/bisque05-caffe-xenial:flour-tg2.2. You can leave it but it's kind of long. Once the build is done, verify that BisQue comes up on your local system just as above:

docker run -itp 8080:8080  --name bqdev-tg2 amilworks/bisque-tg2-upgrade:git

You should see BisQue on http://localhost:8080/ in your browser. If not, paste both the Build and Run Logs below and we will take a look.

Step 3. Make Changes/Break Stuff

GOAL

[ ] Upgrade TurboGears to TurboGears2==2.3.0b2, fix problems that arise, document changes.

Create a new branch Creating a new branch will enable you to make changes, nuke your own branch over and over again, and maintain a clean copy of the original branch when you need to start fresh for the millionth time.

git checkout -b [name_of_your_new_branch]

Open source/requirements.txt with your favorite editor (hopefully Vim). Change the version to the one you are assigned, and save the file.

Here is the link to the line you will update.

TurboGears2==2.2   # <--- Update Version
TurboMail==3.0.3
WebError==0.10.3
WebFlash==0.1a9
WebHelpers==1.3
WebOb==1.1.1       # <--- Potential Update Needed, Check Build Logs

Push to GitHub You made changes and you want to have version control. Push your new branch and changes to GitHub by running:

git push -u origin  [name_of_your_new_branch]

Step 4. Build and Run!

GOAL

[ ] Build and run your changes, fix any dependency issues, document changes, rebuild if necessary (most likely unless you are super lucky), trust the process, and always ask for help if you get stuck!

Once you made your changes, make sure you cd .. out of source/ and are back to the tree view displayed above. You should see a file called Makefile. You can Edit this file as mentioned in Option 2. or simply run the following command to build the image with your updated TurboGears2.

make bisque05-caffe-xenial

While its building, make sure that there are no errors during the pip installs. Typically errors are skipped over in favor of a successful build. I know, right?

Next, once the build is finished, run the image and verify BisQue comes up on your local system:

docker run -itp 8080:8080  --name bqdev-tg2 amilworks/bisque-tg2-upgrade:git

NOTE: There might be other packages that need updating when upgrading to different TurboGears2 versions. Sometimes there will be an error raised with the version required, or it simply will not run and you will have to do a logsdump here and we will have to sift through it together to track the failed step(s).

If everything is successful, push your branch to GitHub if you have not already, file a pull request, and pat yourself on the back. Job well done. We will review it and test it on our production-like setup and give feedback if any issues arise.

Post Upgrade

Use this issue to document any issues you faced during the upgrade process. Log dumps can be thrown here as well. Try to use the code syntax for large blocks of code so we can easily read through it. Otherwise, go crazy.

[x] What broke, if anything? Just a simple list will suffice.
[ ] Were there any files significantly changed or that requires significant attention?
[ ] Can we move forward to the next upgrade?
[ ] Any blockers now or that you foresee?

FeiX-OvO commented 4 years ago

Hey Amil, I was at step 2 but it seems like I have no permission to pull the image: docker pull amilworks/bisque-tg2-upgrade:git Error response from daemon: pull access denied for amilworks/bisque-tg2-upgrade, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

amilworks commented 4 years ago

Yeah turns out the automated build was using the wrong branch. My bad! Should be fixed now.

FeiX-OvO commented 4 years ago

Yeah working now. Thanks!

FeiX-OvO commented 4 years ago

Hi @amilworks I was having a problem with

make bisque05-caffe-xenial

It stops running and gives an error message in the middle:

The command '/bin/sh -c wget -q https://bitbucket.org/dimin/bioimageconvert/downloads/$IMGCNV.tar.gz && tar xf $IMGCNV.tar.gz && mv $IMGCNV/imgcnv /usr/bin && mv $IMGCNV/libimgcnv.so* /usr/lib/x86_64-linux-gnu/ && rm -rf $IMGCNV && apt-get install -y --no-install-recommends libswscale-ffmpeg3 libfftw3-3 libgdcm2.6 libavcodec-ffmpeg56 libavformat-ffmpeg56 libavutil-ffmpeg54 libhdf5-cpp-11' returned a non-zero code: 8 make: *** [bisque05-caffe-xenial] Error 8

I thought it was because of my change, but with the original code, it still comes to the same thing.

FeiX-OvO commented 4 years ago

Also, I'm wondering what's the difference between requirement.txt in and out of the source directory. They seem pretty much the same for me. As I could understand, I think the things in /soucre were what we worked on in the container, so when we updating Turbogears outside it, why do we modify the requiremnts.txt inside the folder?

amilworks commented 4 years ago

There's no difference, just change the one in the source folder.

So try pulling the prebuilt image from the DockerHub repo I mentioned before since it will skip that step. Basically it will mount it from that image.

docker pull amilworks/bisque-tg2-upgrade:git

FeiX-OvO commented 4 years ago

Last time we said I have to exit the container, edit the file, and make it to a new image to build the container. In this way, how should I skip the step?

Or is it just the same to change the .txt in the container built on the image I pulled? Like what we did before.

@amilworks not sure if you saw this...;/

amilworks commented 3 years ago

Oh yeah, so you will have to rebuild the image once you make the changes. So once you change the source/requirements.txt from your local branch, rebuild the image.

FeiX-OvO commented 3 years ago

But then I cannot skip make image...? Sorry I'm super lost...

amilworks commented 3 years ago

I updated the image so it bypasses the Image converter installation. Let me know if it works!

Pull the new changes, try your make command from before, and it should work. I tested it on my workstation so double check on yours.

FeiX-OvO commented 3 years ago

sudo apt update sudo apt install curl

pip install -r requirements.txt

ERROR: Could not find a version that satisfies the requirement Minimatic==1.0.4 (from -r requirements.txt (line 9)) (from versions: 1.0)
ERROR: No matching distribution found for Minimatic==1.0.4 (from -r requirements.txt (line 9))

Minimatic==1.0

ERROR: Could not find a version that satisfies the requirement Paste==1.7.5.2 (from -r requirements.txt (line 10)) (from versions: 0.3, 0.4, 0.4.1, 0.5, 0.9, 0.9.1, 0.9.2, 0.9.3, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 0.9.8.1, 1.0, 1.0.1, 1.1, 1.1.1, 1.2, 1.2.1, 1.3, 1.4, 1.4.1, 1.4.2, 1.5, 1.5.1, 1.6, 1.7, 1.7.1, 1.7.2, 1.7.3, 1.7.3.1, 1.7.4, 1.7.5, 1.7.5.1, 2.0, 2.0.1, 2.0.2, 2.0.3, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.0.5, 3.0.6, 3.0.7, 3.0.8, 3.1.0, 3.1.1, 3.2.0, 3.2.1, 3.2.2, 3.2.3, 3.2.4, 3.2.5, 3.2.6, 3.2.7, 3.3.0, 3.4.0, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.4.5, 3.4.6, 3.5.0)
ERROR: No matching distribution found for Paste==1.7.5.2 (from -r requirements.txt (line 10))

Paste==1.7.5.1

ERROR: Could not find a version that satisfies the requirement TurboGears2==2.3.0b2 (from -r requirements.txt (line 23)) (from versions: 2.0.4, 2.1, 2.1.1, 2.1.2, 2.1.3, 2.1.4, 2.1.5, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 2.3.7, 2.3.8, 2.3.9, 2.3.10, 2.3.11, 2.3.12, 2.4.0a1, 2.4.0a2, 2.4.0, 2.4.1, 2.4.2, 2.4.3)
ERROR: No matching distribution found for TurboGears2==2.3.0b2 (from -r requirements.txt (line 23))

TurboGears2==2.3.0

ERROR: Could not find a version that satisfies the requirement httplib2==0.7.1-1 (from -r requirements.txt (line 52)) (from versions: 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.8, 0.9, 0.9.1, 0.9.2, 0.10.3, 0.11.0, 0.11.1, 0.11.3, 0.12.0, 0.12.1, 0.12.3, 0.13.0, 0.13.1, 0.14.0, 0.15.0, 0.16.0, 0.17.0, 0.17.1, 0.17.2, 0.17.3, 0.17.4, 0.18.0, 0.18.1)
ERROR: No matching distribution found for httplib2==0.7.1-1 (from -r requirements.txt (line 52))

httplib2==0.7.3

ERROR: Could not find a version that satisfies the requirement tgext.registration2==0.5.2 (from -r requirements.txt (line 91)) (from versions: none)

pip install registration==0.50 ?

pip install tgapp-registration?

ERROR: Could not find a version that satisfies the requirement tw.output==0.5.0dev-20110906 (from -r requirements.txt (line 95)) (from versions: none)

https://turbogears.org/cogbin.html

urllib2.HTTPError: HTTP Error 403: SSL is required
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

pydicom

pip install psycopg2==2.6.1

Please add the directory containing pg_config to the PATH
    or specify the full executable path with the option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

pip install psycopg2-binary earliest version 2.7.3.2

pip install pysqlite

fatal error: Python.h: No such file or directory

sudo apt-get install python-dev

graphviz

sudo apt-get install python-dev graphviz libgraphviz-dev pkg-config

networkx==1.7

FeiX-OvO commented 3 years ago

In Makefile:

Generating packaged JS and CSS files

Traceback (most recent call last): File "/usr/lib/bisque/bin/bq-admin", line 11, in load_entry_point('bqcore', 'console_scripts', 'bq-admin')() File "/source/bqcore/bq/commands/admin.py", line 90, in main command.run() File "/source/bqcore/bq/commands/admin.py", line 203, in run self.deploy_public() File "/source/bqcore/bq/commands/admin.py", line 300, in deploy_public generate_css_files(root=rootdir, public=publicdir) File "/source/bqcore/bq/core/lib/js_includes.py", line 76, in generate_css_files *css_kw File "/usr/lib/bisque/local/lib/python2.7/site-packages/minimatic/init.py", line 382, in stylesheet_link return base_link('css', sources, **options) File "/usr/lib/bisque/local/lib/python2.7/site-packages/minimatic/init.py", line 347, in base_link touch(combined_path) File "/usr/lib/bisque/local/lib/python2.7/site-packages/minimatic/init.py", line 48, in touch fhandle = open(fname, 'a') IOError: [Errno 2] No such file or directory: './public/core/css/all_css.css'

FeiX-OvO commented 3 years ago

In generating container:

from tg.controllers import CUSTOM_CONTENT_TYPE ImportError: cannot import name CUSTOM_CONTENT_TYPE

FeiX-OvO commented 3 years ago

edit:

WebOb==1.2 networkx==1.7 Minimatic==1.0

In Make: Could not fetch URL https://biodev.ece.ucsb.edu/py/bisque/xenial/+simple/babel/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='biodev.ece.ucsb.edu', port=443): Max retries exceeded with url: /py/bisque/xenial/+simple/babel/ (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)) - skipping Collecting Babel==1.3 Downloading Babel-1.3.tar.gz (3.4 MB) WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)': /py/bisque/xenial/+simple/beaker/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)': /py/bisque/xenial/+simple/beaker/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)': /py/bisque/xenial/+simple/beaker/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)': /py/bisque/xenial/+simple/beaker/ WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)'),)': /py/bisque/xenial/+simple/beaker/

FeiX-OvO commented 3 years ago

edit: -e git://github.com/TurboGears/tg2.git@tg2.3.0b2#egg=TurboGears2 networkx==1.7 WebOb==1.2 Minimatic==1.0.4

Get error: command.run() File "/source/bqcore/bq/commands/admin.py", line 203, in run self.deploy_public() File "/source/bqcore/bq/commands/admin.py", line 300, in deploy_public generate_css_files(root=rootdir, public=publicdir) File "/source/bqcore/bq/core/lib/js_includes.py", line 76, in generate_css_files *css_kw File "/usr/lib/bisque/local/lib/python2.7/site-packages/minimatic/init.py", line 382, in stylesheet_link return base_link('css', sources, **options) File "/usr/lib/bisque/local/lib/python2.7/site-packages/minimatic/init.py", line 347, in base_link touch(combined_path) File "/usr/lib/bisque/local/lib/python2.7/site-packages/minimatic/init.py", line 48, in touch fhandle = open(fname, 'a') IOError: [Errno 2] No such file or directory: './public/core/css/all_css.css'

github-actions[bot] commented 2 years ago

Stale issue message

UCSB-VRL / bisqueUCSB

TurboGears 2.3.0b2 #13

TurboGears 2.3.0b2

Getting Started

Step 1. Clone the Repo

GOAL

Step 2. Verify Everything Works

GOAL

Step 3. Make Changes/Break Stuff

GOAL

Step 4. Build and Run!

GOAL

Post Upgrade

pydicom