3rd simpler (unweighted only) method of projection in addition to PCA and NMF

[carlocolantuoni]

given an unweighted gene cart (simply a list of genes) this would calculate the number of genes in the cart that are expressed at non-zero levels in each of the samples/cells of the new data set. so the sacle of the projected values wold be zero up to the number of genes in the cart. it wold be a 3rd option where we currently have PCA and NMF as options for projection. we should note that PCA can be for weighted or unweighted, NMF just for weighted and this binary method, just for unweighted. maybe this could be the text: "Binary gene count (unweighted gene carts only)". this would give sers a very easy and straight forward way to use unweighted gene carts across datasets.

[adkinsrs]

Implemented this on my local machine. Code is committed but not pushed to the Github repo yet.

This will require an update to the cloud run service as I changed the "is_pca" input argument to "algorithm" (since we now have 3 algorithms to consider). I am exploring ways we could use two separate revisions of the cloud run service at the same time, so we do not break things when we want to push updates (to gEAR and such), though I think the best solution would be to create a third "staging" or "devel" cloud run service to stage updates before pushing to production.

I also noticed that the number of loadings genes for unweighted genecarts was way too high, because the geardb "get_by_gene_symbol" function had no filter option by organism_id, which I added as an optional input. I don't think this affected the projection plots, since duplicates were removed (post orthology-mapping) before projecting, but it did affect the number that is stored in the projections.json file (which is reported back to the user).

[carlocolantuoni]

great!

On Tue, Feb 7, 2023 at 12:19 PM Shaun Adkins @.***> wrote:

Implemented this on my local machine. Code is committed but not pushed to the Github repo yet.

This will require an update to the cloud run service as I changed the "is_pca" input argument to "algorithm" (since we now have 3 algorithms to consider). I am exploring ways we could use two separate revisions of the cloud run service at the same time, so we do not break things when we want to push updates (to gEAR and such), though I think the best solution would be to create a third "staging" or "devel" cloud run service to stage updates before pushing to production.

I also noticed that the number of loadings genes for unweighted genecarts was way too high, because the geardb "get_by_gene_symbol" function had no filter option by organism_id, which I added as an optional input. I don't think this affected the projection plots, since duplicates were removed (post orthology-mapping) before projecting, but it did affect the number that is stored in the projections.json file (which is reported back to the user).

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1421147381, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7QCLPXUTMBVOGHRTKTWWJ7ZRANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

[adkinsrs]

Built an updated projectR cloud run service and deployed it. Fortunately, I learned that the tags you assign to a revision can be prepended to the URL, which means we can divert "devel" code to a specific tag, and production code to a staging tag. The Google Cloud Run service even has a mode that auto-tags the latest revision, so we can use that for devel servers, and freeze a production tag on the service revision we want to use there

Also pushed commits to the repository (see commit 2eabd3438adf09f1f24c8f2cedc0eeb084ae5480)

[adkinsrs]

Seems to work on nemo-devel, though I have no clue what is the difference between the unweighted gene carts.

[carlocolantuoni]

Great, ill test it out too! Thnx!

On Wed, Feb 8, 2023, 15:02 Shaun Adkins @.***> wrote:

Seems to work on nemo-devel, though I have no clue what is the difference between the unweighted gene carts.

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1423169900, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7VRR7KG5NZTJENK7GDWWP3V3ANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

[carlocolantuoni]

i cant see the 3rd option for projection wen i load transfer learning on nemo devel

On Wed, Feb 8, 2023 at 4:03 PM Carlo Colantuoni @.***> wrote:

Great, ill test it out too! Thnx!

On Wed, Feb 8, 2023, 15:02 Shaun Adkins @.***> wrote:

Seems to work on nemo-devel, though I have no clue what is the difference between the unweighted gene carts.

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1423169900, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7VRR7KG5NZTJENK7GDWWP3V3ANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

[adkinsrs]

It is available when you select an unweighted genecart. Perhaps I should make it visible by default and hide when weighted genecarts are selected

---

Ricky "Shaun" Adkins Bioinformatics Software Engineer II Institute for Genome Sciences University of Maryland, School of Medicine Phone (410) 706-6766

---

From: Carlo Colantuoni @.> Sent: Wednesday, February 8, 2023 10:40 PM To: IGS/gEAR @.> Cc: Adkins, Ricky (Shaun) @.>; Assign @.> Subject: Re: [IGS/gEAR] 3rd simpler (unweighted only) method of projection in addition to PCA and NMF (Issue #487)

i cant see the 3rd option for projection wen i load transfer learning on nemo devel

On Wed, Feb 8, 2023 at 4:03 PM Carlo Colantuoni @.***> wrote:

Great, ill test it out too! Thnx!

On Wed, Feb 8, 2023, 15:02 Shaun Adkins @.***> wrote:

Seems to work on nemo-devel, though I have no clue what is the difference between the unweighted gene carts.

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1423169900, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7VRR7KG5NZTJENK7GDWWP3V3ANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

— Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FIGS%2FgEAR%2Fissues%2F487%23issuecomment-1423577561&data=05%7C01%7Csadkins%40som.umaryland.edu%7C28014ebc41eb423cb56a08db0a4f57f6%7C717009a620de461a88940312a395cac9%7C0%7C0%7C638115108150221680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HJCdKmuOScL0kAyv5%2BAqcZI8tGsgIx%2FPQn3Xm2b3onY%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABLHI6TFMV4N2M3MNGKMADDWWRRJRANCNFSM6AAAAAATSHH3Y4&data=05%7C01%7Csadkins%40som.umaryland.edu%7C28014ebc41eb423cb56a08db0a4f57f6%7C717009a620de461a88940312a395cac9%7C0%7C0%7C638115108150221680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Dp3T2PAn0yTW46qvcXaH99%2Be6xYvJdI3zEUwToEoxeQ%3D&reserved=0. You are receiving this because you were assigned.Message ID: @.***>

[carlocolantuoni]

i cant see it even wen i select an unweighted gene cart

On Wed, Feb 8, 2023 at 10:45 PM Shaun Adkins @.***> wrote:

It is available when you select an unweighted genecart. Perhaps I should make it visible by default and hide when weighted genecarts are selected

---

Ricky "Shaun" Adkins Bioinformatics Software Engineer II Institute for Genome Sciences University of Maryland, School of Medicine Phone (410) 706-6766

---

From: Carlo Colantuoni @.> Sent: Wednesday, February 8, 2023 10:40 PM To: IGS/gEAR @.> Cc: Adkins, Ricky (Shaun) @.>; Assign @.> Subject: Re: [IGS/gEAR] 3rd simpler (unweighted only) method of projection in addition to PCA and NMF (Issue #487)

i cant see the 3rd option for projection wen i load transfer learning on nemo devel

On Wed, Feb 8, 2023 at 4:03 PM Carlo Colantuoni @.***> wrote:

Great, ill test it out too! Thnx!

On Wed, Feb 8, 2023, 15:02 Shaun Adkins @.***> wrote:

Seems to work on nemo-devel, though I have no clue what is the difference between the unweighted gene carts.

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1423169900, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AH7KC7VRR7KG5NZTJENK7GDWWP3V3ANCNFSM6AAAAAATSHH3Y4

. You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

— Reply to this email directly, view it on GitHub< https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FIGS%2FgEAR%2Fissues%2F487%23issuecomment-1423577561&data=05%7C01%7Csadkins%40som.umaryland.edu%7C28014ebc41eb423cb56a08db0a4f57f6%7C717009a620de461a88940312a395cac9%7C0%7C0%7C638115108150221680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=HJCdKmuOScL0kAyv5%2BAqcZI8tGsgIx%2FPQn3Xm2b3onY%3D&reserved=0>, or unsubscribe< https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABLHI6TFMV4N2M3MNGKMADDWWRRJRANCNFSM6AAAAAATSHH3Y4&data=05%7C01%7Csadkins%40som.umaryland.edu%7C28014ebc41eb423cb56a08db0a4f57f6%7C717009a620de461a88940312a395cac9%7C0%7C0%7C638115108150221680%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Dp3T2PAn0yTW46qvcXaH99%2Be6xYvJdI3zEUwToEoxeQ%3D&reserved=0

. You are receiving this because you were assigned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1423579914, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7WS6DBUFMTVRSLX23TWWRR37ANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

[adkinsrs]

So you aren't seeing this Binary Gene Count option? Is there a certain order you are selecting the genecart and profile?

[carlocolantuoni]

i did not see it come up til now! but yes now i do see it! crazy! sorry for the run-around. testing a bit now

On Wed, Feb 8, 2023 at 11:12 PM Shaun Adkins @.***> wrote:

[image: Screenshot 2023-02-08 at 11 11 34 PM] https://user-images.githubusercontent.com/5665914/217715932-bc7170ae-40bc-421b-874d-8fe3f0a37e36.png

So you aren't seeing this? Is there a certain order you are selecting the genecart and profile?

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1423603598, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7WD6KZWVSBDAS6Y5LTWWRVDZANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

[carlocolantuoni]

in general it appears to be working, but im getting a good number of these "There was an error projecting patterns onto this dataset."

On Wed, Feb 8, 2023 at 11:16 PM Carlo Colantuoni @.***> wrote:

i did not see it come up til now! but yes now i do see it! crazy! sorry for the run-around. testing a bit now

On Wed, Feb 8, 2023 at 11:12 PM Shaun Adkins @.***> wrote:

[image: Screenshot 2023-02-08 at 11 11 34 PM] https://user-images.githubusercontent.com/5665914/217715932-bc7170ae-40bc-421b-874d-8fe3f0a37e36.png

So you aren't seeing this? Is there a certain order you are selecting the genecart and profile?

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1423603598, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7WD6KZWVSBDAS6Y5LTWWRVDZANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

-- Carlo

[carlocolantuoni]

those errors seem to have gone away, so looking really good! on one of the datasets that never load de to gen id issues, im getting a new error -"Could not remap pattern genes to ortholog equivalent in the dataset" guessing this is something joshua pt in place as he started doing the gene id work? just fyi carlo

On Wed, Feb 8, 2023 at 11:27 PM Carlo Colantuoni @.***> wrote:

in general it appears to be working, but im getting a good number of these "There was an error projecting patterns onto this dataset."

On Wed, Feb 8, 2023 at 11:16 PM Carlo Colantuoni < @.***> wrote:

i did not see it come up til now! but yes now i do see it! crazy! sorry for the run-around. testing a bit now

On Wed, Feb 8, 2023 at 11:12 PM Shaun Adkins @.***> wrote:

[image: Screenshot 2023-02-08 at 11 11 34 PM] https://user-images.githubusercontent.com/5665914/217715932-bc7170ae-40bc-421b-874d-8fe3f0a37e36.png

So you aren't seeing this? Is there a certain order you are selecting the genecart and profile?

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1423603598, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7WD6KZWVSBDAS6Y5LTWWRVDZANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

-- Carlo

-- Carlo

[adkinsrs]

"Could not remap pattern genes to ortholog equivalent in the dataset" means the orthology feature mapping failed.

[adkinsrs]

in general it appears to be working, but im getting a good number of these "There was an error projecting patterns onto this dataset." On Wed, Feb 8, 2023 at 11:16 PM Carlo Colantuoni @.> wrote: … i did not see it come up til now! but yes now i do see it! crazy! sorry for the run-around. testing a bit now On Wed, Feb 8, 2023 at 11:12 PM Shaun Adkins @.> wrote: > [image: Screenshot 2023-02-08 at 11 11 34 PM] > https://user-images.githubusercontent.com/5665914/217715932-bc7170ae-40bc-421b-874d-8fe3f0a37e36.png > > So you aren't seeing this? Is there a certain order you are selecting the > genecart and profile? > > — > Reply to this email directly, view it on GitHub > <#487 (comment)>, or > unsubscribe > https://github.com/notifications/unsubscribe-auth/AH7KC7WD6KZWVSBDAS6Y5LTWWRVDZANCNFSM6AAAAAATSHH3Y4 > . > You are receiving this because you were assigned.Message ID: > @.***> > -- Carlo -- Carlo

Regarding this, I just discovered that the rabbitmq service was turned off in the configs, so that would explain this particular error. However when I start it back up I am getting errors with the consumer (which is weird since I can get it to work on gear-devel). I'm looking into this.

[adkinsrs]

RE: previous comment... I fixed the rabbitmq consumer error on nemo-devel and things seem to be running now in "rabbitmq mode". I discovered the existing "projectr" consumer queue was created under a different configuration, and this conflicted with the current consumer queue I was trying to create, so I had to wipe out the original one.

[carlocolantuoni]

👍

On Thu, Feb 9, 2023 at 1:27 PM Shaun Adkins @.***> wrote:

RE: previous comment... I fixed the rabbitmq consumer error on nemo-devel and things seem to be running now in "rabbitmq mode". I discovered the existing "projectr" consumer queue was created under a different configuration, and this conflicted with the current consumer queue I was trying to create, so I had to wipe out the original one.

— Reply to this email directly, view it on GitHub https://github.com/IGS/gEAR/issues/487#issuecomment-1424634751, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH7KC7V6D277WDQOOYRKHQDWWUZJNANCNFSM6AAAAAATSHH3Y4 . You are receiving this because you were assigned.Message ID: @.***>

-- Carlo

[adkinsrs]

This is implemented so I am closing