search

biocore / songbird

Vanilla regression methods for microbiome differential abundance analysis
BSD 3-Clause "New" or "Revised" License
54 stars 25 forks source link

Using songbird standalone #134

Open coralzhang opened 3 years ago

coralzhang commented 3 years ago

Hi,

I am using a Mac Catelina This is what I did when installing

conda create -n songbird_env python=3.6 numpy=1.15.4 scikit-bio=0.5.5 seaborn pandas=0.23.4 -c conda-forge
source activate songbird_env
conda install tensorflow=1.30 tqdm nomkl
conda install biom-format h5py -c conda-forge
conda install jupyter notebook
conda install songbird -c conda-forge

I keep getting the following warnings when I run model as simple as this

songbird multinomial \
    --input-biom ./data/redsea.biom \
    --metadata-file ./data/redsea_metadata.txt \
    --formula "1" \
    --epochs 10000 \
    --differential-prior 0.5 \
    --training-column Testing \
    --summary-interval 1 \
    --summary-dir results-redsea/redsea-results-null

Below are warnings

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/bin/songbird:145: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-08-05 13:59:42.889984: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags. 2020-08-05 13:59:42.890306: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 8. Tune using inter_op_parallelism_threads for best performance. WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:75: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.random.categorical instead. WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:86: The name tf.random_normal is deprecated. Please use tf.random.normal instead.

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:91: Normal.init (from tensorflow.python.ops.distributions.normal) is deprecated and will be removed after 2019-01-01. Instructions for updating: The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use tfp.distributions instead of tf.distributions. WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/tensorflow/python/ops/distributions/normal.py:160: Distribution.init (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01. Instructions for updating: The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use tfp.distributions instead of tf.distributions. WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:100: Multinomial.init (from tensorflow.python.ops.distributions.multinomial) is deprecated and will be removed after 2019-01-01. Instructions for updating: The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use tfp.distributions instead of tf.distributions. WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:115: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:121: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/tensorflow/python/ops/clip_ops.py:286: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:129: The name tf.summary.histogram is deprecated. Please use tf.compat.v1.summary.histogram instead.

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:130: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:132: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:136: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:163: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

0%| | 0/80000 [00:00<?, ?it/s]WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:172: The name tf.RunOptions is deprecated. Please use tf.compat.v1.RunOptions instead.

WARNING:tensorflow:From /usr/local/anaconda3/envs/songbird_env/lib/python3.6/site-packages/songbird/multinomial.py:174: The name tf.RunMetadata is deprecated. Please use tf.compat.v1.RunMetadata instead.

In addition, when I try to load tensorboard, I keep getting these warnings (which goes longer than this but similar) tensorboard --logdir .

I0805 14:05:20.793066 123145466290176 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:20] "GET / HTTP/1.1" 200 - I0805 14:05:21.125150 123145461035008 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:21] "GET /font-roboto/oMMgfZMQthOryQo9n22dcuvvDin1pK8aKteLpeZ5c0A.woff2 HTTP/1.1" 200 - I0805 14:05:21.136273 123145471545344 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:21] "GET /font-roboto/RxZJdnzeo3R5zSexge8UUZBw1xU1rKptJj_0jans920.woff2 HTTP/1.1" 200 - I0805 14:05:21.153836 123145476800512 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:21] "GET /font-roboto/d-6IYplOFocCacKzxwXSOJBw1xU1rKptJj_0jans920.woff2 HTTP/1.1" 200 - I0805 14:05:21.648594 123145482055680 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:21] "GET /data/environment HTTP/1.1" 200 - I0805 14:05:21.652559 123145487310848 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:21] "GET /data/runs HTTP/1.1" 200 - I0805 14:05:21.654026 123145492566016 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:21] "GET /data/experiments HTTP/1.1" 200 - I0805 14:05:21.677382 123145466290176 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:21] "GET /data/plugins_listing HTTP/1.1" 200 - I0805 14:05:21.824388 123145461035008 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:21] "GET /data/plugin/scalars/tags HTTP/1.1" 200 - I0805 14:05:22.475150 123145471545344 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:22] "GET /data/plugin/scalars/scalars?tag=loss&run=.&experiment= HTTP/1.1" 200 - I0805 14:05:22.479615 123145461035008 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:05:22] "GET /data/plugin/scalars/scalars?tag=accuracy%2Fcv_error&run=.&experiment= HTTP/1.1" 200 -

I cannot load the HPARAMS page, which is just empty... Here are the warnings when I try to load hparams

E0805 14:09:00.260663 123145461035008 hparams_plugin.py:108] HParams error: Can't find an HParams-plugin experiment data in the log directory. Note that it takes some time to scan the log directory; if you just started Tensorboard it could be that we haven't finished scanning it yet. Consider trying again in a few seconds. I0805 14:09:00.262099 123145461035008 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:09:00] "POST /data/plugin/hparams/experiment HTTP/1.1" 400 - I0805 14:09:00.263484 123145466290176 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:09:00] "GET /data/plugins_listing HTTP/1.1" 200 - E0805 14:09:00.280741 123145461035008 hparams_plugin.py:108] HParams error: Can't find an HParams-plugin experiment data in the log directory. Note that it takes some time to scan the log directory; if you just started Tensorboard it could be that we haven't finished scanning it yet. Consider trying again in a few seconds. I0805 14:09:00.280971 123145461035008 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:09:00] "POST /data/plugin/hparams/experiment HTTP/1.1" 400 - E0805 14:09:00.289092 123145461035008 hparams_plugin.py:108] HParams error: Can't find an HParams-plugin experiment data in the log directory. Note that it takes some time to scan the log directory; if you just started Tensorboard it could be that we haven't finished scanning it yet. Consider trying again in a few seconds. I0805 14:09:00.289318 123145461035008 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:09:00] "POST /data/plugin/hparams/experiment HTTP/1.1" 400 - E0805 14:09:00.324394 123145461035008 hparams_plugin.py:108] HParams error: Can't find an HParams-plugin experiment data in the log directory. Note that it takes some time to scan the log directory; if you just started Tensorboard it could be that we haven't finished scanning it yet. Consider trying again in a few seconds. I0805 14:09:00.324654 123145461035008 _internal.py:113] ::ffff:192.168.0.184 - - [05/Aug/2020 14:09:00] "POST /data/plugin/hparams/experiment HTTP/1.1" 400 -

This is what I see image

mortonjt commented 3 years ago

hmm. We'll need more information in order to debug. How did you run the tensorboard command? Did you run everything in the same folder? Do you have any files generated in your results folder?

coralzhang commented 3 years ago

Thanks for the response! Here are responses. How did you run the tensorboard command?

Yes and this is what I did tensorboard --logdir .

Did you run everything in the same folder?

Yes. I am staying in the same big folder except go in and out of the result folder...

Do you have any files generated in your results folder?

Yes and saved here https://www.dropbox.com/sh/3sxl94dh9xhirt5/AADLzmVwihtaAvb9IYyjezV4a?dl=0

I am not sure but I think it may have a relationship with the tensorflow version? when i install and use 1.10. Everything goes fine; the only issue is that I cannot see the HPARAMS tab, which was not available in that version.

mortonjt commented 3 years ago

Got it. Have you tried Tensorboard version 1.15.2?

On Wed, Aug 5, 2020, 3:34 PM coralzhang notifications@github.com wrote:

Thanks for the response! Here are responses. How did you run the tensorboard command?

Yes and this is what I did tensorboard --logdir .

Did you run everything in the same folder?

Yes. I am staing in the same folder.

Do you have any files generated in your results folder?

Yes and saved here https://www.dropbox.com/sh/3sxl94dh9xhirt5/AADLzmVwihtaAvb9IYyjezV4a?dl=0

I am not sure but I think it may have a relationship with the tensorflow version? when i install and use 1.10. Everything goes fine; the only issue is that I cannot see the HPARAMS tab, which was not available in that version.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/biocore/songbird/issues/134#issuecomment-669524275, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA75VXOTVMJNFXXBHJ2G36DR7HF7FANCNFSM4PV4TZ3Q .

coralzhang commented 3 years ago

That is part of my question.. How do I upgrade tensorboard itself properly? I have never used conda and tensor flow before. As you see, the way I installed tensorborad, I guess, is through installing of tensorflow version=xx, and I tried to simply uninstall tensorboard and upgrade, but end up with messing up stuff with tb-nightly as well. So, I then decided to use a higher version of tensorflow, say 1.15. Then got here. In addition, since the key results are the differentials.tsv, if I run a multinomial regression somewhere else, will I be able to reproduce the results, at least largely?

mortonjt commented 3 years ago

Try conda install tensorboard=1.15.2 There were no breaking changes between versions, so you should be able to reproduce the results

On Wed, Aug 5, 2020, 3:48 PM coralzhang notifications@github.com wrote:

That is part of my question.. How do I upgrade tensorborad itself properly? I have never used conda and tensor flow before. As you see, the way I installed tensorborad, I guess, is through installing of tensorflow version=xx, and I tried to simply uninstall tensorboard and upgrade, but end up with messing up stuff with tb-nightly as well. So, I end up trying to use a higher version of tensorflow, say 1.15. In addition, since the key results are the differentials.tsv, if I run a multinomial regression somewhere else, will I be able to reproduce the results, at least largely?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/biocore/songbird/issues/134#issuecomment-669530974, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA75VXNU4FZLUPYM2D4Y7R3R7HHUPANCNFSM4PV4TZ3Q .