Open FedorSteeman opened 2 years ago
FedorSteeman
commented
2 years ago The first phase is creating a test environment to try the waters. For this purpose, KU-IT has created new virtual servers in the SCIENCE domain with the following names:
FedorSteeman
commented
2 years ago Ordered server access to @jlegind so he can take over the DaSSCo app bit.
FedorSteeman
commented
2 years ago Downloaded Specify7 as both current (7.5.0) and latest versions (7.6.1). Still cannot get the latest to compile. May attempt with Docker.
Cloned sputil and prepared apache for serving it as a web app via url: sputil.science.ku.dk Created KU IT support tickets for opening ports and adding this subdomain.
Note: Apparently, when in office, it's not necessary to use the Cisco VPN client to be able log in to these servers.
FedorSteeman
commented
2 years ago Progress:
FedorSteeman
commented
2 years ago Status:
FedorSteeman
commented
2 years ago Ordered subdomain specify-nhmd.science.ku.dk
FedorSteeman
commented
2 years ago In a sudden twist, I unexpectedly find myself able to compile specify7.6.1 (i.e. the latest version) on the new as well as the old web servers. This means that, if the database can likewise be upgraded, we can already move Specify to the next level without having to wait on the creation of a new server suite in the new KU environment.
FedorSteeman
commented
2 years ago Discussed upcoming needs with KU-ITs Erik Sværke. We agreed that the future Specify setup would at least need 4 servers:
I changed the order for subdomain specify-nhmd.science.ku.dk to specify-dev.science.ku.dk
FedorSteeman
commented
2 years ago Successfully compiled 7.6.1 on all servers, so they're ready for the upgrade. Note: I used the following sequence of instructions to do a successful compilations: specify7-serverupgrade-bash.txt
Next step is upgrading (a copy of) each database for each institution and see if the upgraded version works for the the end users, in close dialogue. I will make sure to be able to rollback in case of troubles.
I will start with the smaller museums first and leave NHMD at the end.
FedorSteeman
commented
2 years ago Successfully upgraded https://specify-oesm.science.ku.dk/specify/
However, some tweaking of the bash commands were required that I'm gathering here: https://github.com/NHMDenmark/DanSpecify/blob/master/Scripts/Server/specify7-serverupgrade-bash.txt
FedorSteeman
commented
2 years ago Successfully upgraded https://specify-naturama.science.ku.dk/specify/
While doing the upgrade, I discovered that the SSL certificates were not yet setup using LetsEncrypt, so fixed that and added notes of my process here: https://github.com/NHMDenmark/DanSpecify/blob/master/Scripts/Server/specify7-letsencrypt-troubleshooting.txt
FedorSteeman
commented
2 years ago Reminder to my self: FIMUS is also not set up with LetsEncrypt yet.
FedorSteeman
commented
2 years ago The upgrade of https://specify-naturama.science.ku.dk/specify/ ran into some unexpected problems that I reported here: https://github.com/specify/specify7/issues/1743
FedorSteeman
commented
2 years ago Problems resolved by making sure the python models were migrated.
Successfully upgraded https://specify-fimus.science.ku.dk/specify/
FedorSteeman
commented
2 years ago Successfully upgraded:
Waiting for possible feedback from any end users.
FedorSteeman
commented
2 years ago Successfully upgraded https://specify-muserum.science.ku.dk/specify/
Waiting for possible feedback from any end users.
FedorSteeman
commented
2 years ago Successfully upgraded https://specify-snm.science.ku.dk/specify/
All upgrades done!
FedorSteeman
commented
1 year ago I have not been logging progress, but here's a short summary:
I will continue getting Specify7 up and running, and when succesful, initiate the move of the SNM installation, preferably during summer break.
FedorSteeman
commented
1 year ago The new issue is with the web asset server running on specifyapp01fl that cannot currently be accessed externally.
I've checked whether the service is running by trying to access the localhost/web_asset_store.xml and it works.
Created a ticket with KU-IT to open for ports 80, 8080 and 443
FedorSteeman
commented
1 year ago The web server (specifyweb01fl) can now access the app server (specifyapp01fl) from where the web asset server runs.
The next issue was with the web server connecting with the database server which returns an "access denied" for the "Master" user and even for the root user. Adjusting the host access settings for each user appeared to alleviate this issue, temporarily at least.
The Specify7 finally sprung to live using the following URL: http://specifyweb01fl.unicph.domain/specify/
The following URL can also be used, since the subdomain had already been ordered in advance: http://specify-nhmd.science.ku.dk/specify/
There was though an issue with the web asset server not working properly. First of all, it still needed a mount to an external server holding all the media, but one folder (for the Biocultural Botany collection) had been pre-copied to the server itself. So in principle it should be possible for these particular images to be served.
The following sample URL should then give results: http://specifyapp01fl.unicph.domain/fileget?coll=NHMD+Vascular+Plants&type=T&filename=sp6999962999599351585.att.jpg&downloadname=NHMD-846654.jpg
The link doesn't work from a browser, so I created a new ticket at KUIT for having ports 80 (&443) opened to the general internet for the app server. The port is open from the web server to the app server, so running a GET from the server prompt should work.
However, this returned a sever error. It appeared from the access log that only the first parameter was transferred and not the subsequent ones. Using a trick by putting the URL in a text file did make sure the URL arrives intact.
The next problem is that the new version of the web asset server expects a scale parameter, so this was added as follows:
This resulted into an actual download of the jpg file onto the server. So far so good...
Next thing is copying all the attachments to a drive that can be mounted to the web asset server.
An odd thing that happened is that the "access denied" for the database root user returned for unknown reasons...
FedorSteeman
commented
1 year ago The access denied error has been fixed.
KU-IT has been stalling and still haven't opened the firewall port for the web asset server.
FedorSteeman
commented
1 year ago Not sure if this is related to the changes I made to settings, but the web asset server is now giving 502 errors:
http://specifyapp01fl.unicph.domain/web-asset-store.xml
In the log the following errors is typically reported:
connect() failed (111: Connection refused) while connecting to upstream, client: 10.175.116.37, server: specifyapp01fl.unicph.domain, request: "GET /web-asset-store.xml HTTP/1.1", upstream: "http://127.0.0.1:8080/web-asset-store.xml", host: "specifyapp01fl.unicph.domain"
FedorSteeman
commented
1 year ago Things are moving forward. It appeared not to be a firewall issue, but a software issue. I reached out to Kansas via Discourse and they have put out some fixes that are making progress, but we're not quite there yet.
https://discourse.specifysoftware.org/t/specify7-in-docker-keeps-on-failing-on-port-8080/1361
To mitigate the many performance issues and the current inability to upgrade to the latest version of Specify, plans have been made to gradually move all servers to the new SCIENCE infrastructure.