Onboarding for Cecilia & A’Kaia

[dondi]

@ceciliazaragoza and @akaiap can now follow the onboarding wiki page (https://github.com/dondi/GRNsight/wiki/Onboarding-Checklist) to get themselves set up w/ GRNsight. @ntran18 and @nchun2 can be consulted in case any issues come up; same with @dondi if there’s anything particularly thorny

Also review the GRNsight publications page (https://dondi.github.io/GRNsight/publications.html) starting with the 2016 PeerJ paper and reviewing some posters in order to start familiarizing yourselves with the application domain

[ceciliazaragoza]

Hi everyone, here are a few changes I think we should make to the documentation for Initial Setup.

1. GRNsight/database/README.md Pt. 1
   • Install the software at this link section:
   • For MacOS users, it is recommended to install with homebrew rather than the interactive installation in order to correctly view the initdb --locale=C -E UTF-8 location-of-cluster message in the documentation.
   • Start and stop the server section:
   • After installing with homebrew on MacOS, you may receive an error when you try to start the server that the server is unable to be started, and when attempting to stop the server, there terminal states there is no server running. In this case, you have to directly kill the port that the server is running on.
   • To double check that this is the issue, you can open the Activity Monitor app on your computer and search for the postgres activity. If there is one, that means the server is running, and we have to terminate the port that the server is running on.
   • First, we have to check what port the server is running on. Navigate to your homebrew installation, which is the same location-of-cluster from when the database was initialized and open that location in VSCode.
   • Search for port = in the file postgresql.conf. By default, the port should be port 5432, but keep note of this port in case it is different.
   • Refer to this Stack Overflow documentation on how to kill a server:
     • https://stackoverflow.com/questions/4075287/node-express-eaddrinuse-address-already-in-use-kill-server
   • If that doesn't work, then refer to the different methods on this link from Stack Overflow:
     • https://stackoverflow.com/questions/42416527/postgres-app-port-in-use
2. https://github.com/dondi/GRNsight/wiki/Initial-Setup#database-setup Database setup
   • On MacOs if you receive the error in your terminal psql: FATAL: role "postgres" does not exist, then try this solution from Stack Overflow(https://stackoverflow.com/questions/15301826/psql-fatal-role-postgres-does-not-exist?page=2&tab=scoredesc#tab-top):
   • sudo -u postgres createuser -s postgres

[kdahlquist]

@ceciliazaragoza is finished with the initial setup. @akaiap is still in progress. She is having issues with the PPI database and plans to attend @dondi's office hours. @akaiap will also add herself to the People page and should just edit the gh-pages branch directly. They have reviewed the paper/posters/slides about GRNsight.

[dondi]

@akaiap and @dondi worked on the database loading issues and found some incompatibilities with her setup for which we did identify solutions. However, what remains to be done is to track down exactly where the incompatibility exists (e.g., is it Python version? OS version? etc.?)

Here are the links that describe the errors we encountered, along with fixes:

• urlparse issue: https://github.com/intermine/intermine-ws-python/issues/96 and https://github.com/intermine/intermine-ws-python/pull/97
• SSL_CERTIFICATE_VERIFY_FAILED issue: https://stackoverflow.com/questions/35569042/ssl-certificate-verify-failed-with-python3 —for this solution, the fix needed to be placed in the same webservice.py file as the Interline GitHub PR above

Based on the issues and their solutions, it’s looking like Intermine may be falling behind in keeping up with the latest versions of Python. However, @akaiap will investigate this with @ceciliazaragoza and @ntran18 to pinpoint the exact software differences that reveal these incompatibilities

[dondi]

@akaiap is running into an unexpected issue where npm install doesn’t appear to be creating a _nodemodules folder as it should. This in turn may be leading to the command not found errors because the system can’t find the installed software

@dondi will need to investigate this further and come back with additional things to try

[dondi]

After additional investigation, @akaiap observed that the node_modules folder does appear but is subsequently deleted. This Stack Overflow question hints that iCloud Sync may be the culprit

@akaiap will try deactivating iCloud Sync once she has made a backup of her files, just in case something does wrong. Backup can be via Box if there is enough space, or also an external drive

[akaiap]

I moved the files from iCloud Sync-- to the machine. After running npm update, install and run start dev. A new error was given. `> parallelshell --wait "nodemon web-client/app.js" "nodemon server/app.js" "webpack --watch --mode=development"

sh: parallelshell: command not found npm ERR! code ELIFECYCLE npm ERR! syscall spawn npm ERR! file sh npm ERR! errno ENOENT npm ERR! grnsight@6.0.7 start-dev: parallelshell --wait "nodemon web-client/app.js" "nodemon server/app.js" "webpack --watch --mode=development" npm ERR! spawn ENOENT npm ERR! npm ERR! Failed at the grnsight@6.0.7 start-dev script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above`. Will work with @dondi to fix this error.

[dondi]

“Command not found” still implies to me that the software isn’t fully installed, and hints that the observed deletion might still be taking place. The comment says that the files were moved, but didn’t say that iCloud Sync was actually turned off. @akaiap is iCloud Sync fully deactivated? (again with the reminder to make sure you have a reliable backup of your files before you do this)

[akaiap]

After fully deactivating the iCloud Sync and creating a backup for the files, I reinstalled npm, and ran the command npm run start-dev. The node modules did not appear, so I ran other files that use npm and I got the same result. sh: parallelshell: command not found npm ERR! code ELIFECYCLE npm ERR! syscall spawn npm ERR! file sh npm ERR! errno ENOENT npm ERR! grnsight@6.0.7 start-dev:parallelshell --wait "nodemon web-client/app.js" "nodemon server/app.js" "webpack --watch --mode=development" npm ERR! spawn ENOENT npm ERR! After getting this error-- I did the steps as done earlier. (Deleted NodeModules, and package-lock.json, the ndid npm install. The install could not finish and I received this after 10 minutes of waiting for it to install npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated ⸨ ░░░░░░░░░░░░░░░⸩ ⠏ fetchMetadata: sill resolveWithNewModule ctype@0.5.3 chec⸨ ░░░░░░░░░░░░░░░⸩ ⠏ fetchMetadata: sill resolveWithNewModule ctype@0.5.3 chec⸨ ░░░░░░░░░░░░░░░⸩ ⠏ fetchMetadata: sill resol⸨ ░░░░░░░░░░░░░░░⸩ ⠏ fetchMeta⸨ ░░░░░░░░░░░░░░░⸩ ⠏ fetchMetadata: sill resolveWithNewModule ctype@0.5.3 chec⸨ ░░░░░░░░░░░░░░░⸩ ⠏ fetchMetadata: sill resolveWithNewModule ctype@0.5.3 chec⸨ ░░░░░░⸩ ⠸ extract:rxjs: sill extract rxjs@^5.5.2 extracted to /Use Recloned the repo and received (new)this error `npm WARN tar ENOENT: no such file or directory, open '/Users/aphelps/GRNsight2/GRNsight/node_modules/.staging/rxjs-3bcfbfee/LICENSE.txt' npm WARN tar ENOENT: no such file or directory, open '/Users/aphelps/GRNsight2/GRNsight/node_modules/.staging/rxjs-3bcfbfee/src/LICENSE.txt' npm WARN notsup Unsupported engine for commander@12.0.0: wanted: {"node":">=18"} (current: {"node":"14.21.3","npm":"6.14.18"}) npm WARN notsup Not compatible with your version of node/npm: commander@12.0.0 npm WARN notsup Unsupported engine for commander@12.0.0: wanted: {"node":">=18"} (current: {"node":"14.21.3","npm":"6.14.18"}) npm WARN notsup Not compatible with your version of node/npm: commander@12.0.0 ...npm ERR! Error while executing: npm ERR! /usr/bin/git ls-remote -h -t ssh://git@github.com/eligrey/FileSaver.js.git npm ERR! npm ERR! git@github.com: Permission denied (publickey). npm ERR! fatal: Could not read from remote repository. npm ERR! npm ERR! Please make sure you have the correct access rights npm ERR! and the repository exists. npm ERR! npm ERR! exited with error code: 128

npm ERR! A complete log of this run can be found in: npm ERR! /Users/aphelps/.npm/_logs/2024-02-20T09_27_59_731Z-debug.log aphelps@MacBook-Air-3 GRNsight % `

[akaiap]

Connected with @dondi and @kdahlquist with updates. Downloaded Malware Bytes and had two detections of potential malware on this machine. Will connect tomorrow on whether to quarantine the (PUP)s.

[dondi]

@akaiap quarantined one of the detected malware items but that still didn’t eliminate the node_modules-deletion behavior. She will re-clone the GRNsight repository and try again; next step is to go ahead and do a backup followed by a clean installation. Before restoring from backup, try an npm install on a fresh GRNsight clone

1. Do a clean install (most extreme case is reformat-and-install)
2. Install Xcode to acquire git
3. Install nodejs
4. Clone GRNsight
5. See if the behavior still persists This will at least verify whether the clean install got rid of the malware. Before restoring from backup, we’ll need to look up how to ensure that restoration doesn’t also restore the malware too:
   • Run malware detectors on the backup copy
   • Instead of a restoration utility, copy files manually
   • Do not restore settings or applications from backup; instead, reinstall or reconfigure manually

[akaiap]

During Spring Break, followed 1-5 steps and the issue still persisted. This week I will upload to the hard drive and had reboot my MacBook for malware.

[dondi]

For our clean-install strategy, the plan is to only back up documents/data since malware most likely exists in the OS and executables. We will reformat the hard drive and reinstall a “factory-fresh” version of the OS and will redownload apps and executables so we will aim to do this in a setting with a stable Internet connection

[akaiap]

After backing up and copying documents/data over to the hard drive. I completed the process and factor reset my machine. However to recover and activate the MacBook, I will need to be connected to a Ethernet cable to complete set up with recovery assistance because it does not recognize “LMU-Wireless”. Working at the IT-Helpdesk, I have access to that cable and will complete setup.

[akaiap]

SUCCESSSS!!!! (I am so so happy rn!) Node_modules is building in the GRNsight download and STAYING!! After factory resetting my computer, and trial and error these last couple of days and the packages have been added and audited. I will reinstall/setup database tomorrow, and I will be good to go!

[dondi]

Great to hear! To wrap things up let’s do the following:

• Post some details about that last package dependency—we just want to make sure that this isn’t something new/unexpected or something that needs to be documented as a genuine extra step
• Reported error w/ initdb was solely because the command was executed on the database cluster that Homebrew had already created— @akaiap chose to stay with this default so can proceed with other setup, skipping initdb

[akaiap]

Finally completed! After a lengthy and challenging process, I have successfully completed onboarding! Throughout the journey, I encountered numerous obstacles that were undeniably frustrating, but we successfully got GRNsight running in development mode.

Here's a summary of the process:

1. Initially, I followed the initial setup steps and proceeded to the Database Setup, where I installed the database with the assistance of @ceciliazaragoza and @dondi. However, upon returning to the Initial Setup, I encountered a perplexing new error.
2. The npm install command failed to create the _nodemodules directory, and even when it did, it would disappear after a few seconds. Despite troubleshooting various potential causes such as iCloud sync, node version compatibility, and reinstalling different node versions, as well as completely re-cloning the repository, none of these efforts resolved the issue.
3. Following a malware scan, one result came back positive. The subsequent step was to perform a factory reset on my machine.
4. After consulting with @dondi and @kdahlquist, we concluded that there was malware present on my Mac, which was interfering with the _nodemodules. After backing up my files on a hard drive, I restarted the Initial Setup process on a completely clean machine.
5. Despite encountering a few more errors along the way, success was finally achieved! GRNsight is now officially up and running on my machine!

This was a process I did not expect to run into, but I am glad I did!

[dondi]

@akaiap will add some details on item 5 from the above comment, in order for the team to determine if this needs further documentation

[akaiap]

Regarding database not starting - After creating a super user with the command createuser --interactive --pwprompt
--- I was able to get past the error --> "2024-04-15 19:04:16.456 PDT [14735] FATAL: role "akaiap" does not exist" ---

However, I am now receiving 2024-04-15 20:50:43.712 PDT [22468] ERROR: relation "settings.grnsettings" does not exist at character 44 2024-04-15 20:50:43.712 PDT [22468] STATEMENT: SELECT grnsettings.expression_dataset FROM settings.grnsettings; This is a very interesting error because the schema is already imported.

Additionally, when using the command psql, I receive 2024-04-16 09:39:44.458 PDT [32872] FATAL: database "akaiaphelps" does not exist psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: database "akaiaphelps" does not exist" --> However I changed my superuser to akaiap.

[kdahlquist]

@akaiap is having issues with Python--she's getting a path error, which she has fixed by using a virtual environment. She tried adding to the path, but it didn't fix the problem, so she's going to ask @dondi.

She thinks that the other database issues have to do with using a virtual environment for Python.

[kdahlquist]

Upon inspection of @akaiap's postgres database, we found that she doesn't have the grnsettings table; she needs to run the schema.

[ceciliazaragoza]

@akaiap We should add documentation for the grnsettings-database because I believe I had that issue when setting up too. The documentation for setting up grnsettings-database is not in the database folder README.md for master or beta even though grnsettings-database exists in both branches.

[akaiap]

Successfully Onboarded GRNsight! (officially!) After encountering the issue regarding the database not being displayed, we met and dove deeper into the problem.

• We went into the database using the command psql postgresql://localhost/postgres and \dn to view the tables/List of Schemas
• @ntran18 identified an absence of table from the list: grnsettings-database!
• We went into the README.md folder of grnsettings-database and followed the steps of the file.
• Now the schema was being shown. settings | akaiaphelps!
• Initiating npm run start-dev resulted in success! YAY!!

Items to note: _- @ceciliazaragoza and I have public | pg_database_owner with "pg_database_owner" as owner, however @ntran18 had "ntran" as owner.

Additionally, we discussed the possibility of a virtual environment being a future problem for other projects. At the moment, my machine's environment is tailored to GRNsight and Python should to my main path sooner rather than later.

@ceciliazaragoza Just pushed the documentation for grnsettings-database to Master! Overall, I am beyond happy to have completed onboarding!