search

kad-ecoli / rna3db

maintain local copy of RNA structure database
0 stars 0 forks source link

marc harary meeting 2020-11-12 minute #12

Open kad-ecoli opened 3 years ago

kad-ecoli commented 3 years ago

[1] As we discussed today, it is not correct to combine all three subsets of SPOT-RNA PDB dataset into one single set. Instead, you should train on the training set (TR1), validate hyperparameters (e.g. which epoch to choose) from validation set (VL1) and perform independent testing on test set (TS1).

https://machinelearningmastery.com/how-to-stop-training-deep-neural-networks-at-the-right-time-using-early-stopping/

As shown in the above figure, if you keep training with more epoch, the training loss will always decrease, but the loss on validation set will not improve, or even get worse, after a given number of epochs. Therefore, you should choose the model from the epoch where the MCC and/or F1 score of validation set is the highest, or the validation loss is the smallest.

  1. Could you show me what is the parameter you use when running mxfold2 train?
marc-harary commented 3 years ago

As I mentioned, I succeeded in running the training script for mxfold2. Since you asked for my configuration, I used the following command: mxfold2 train --model MixC --param model.pth --save-config model.conf data/TR0-canonicals.lst

However, when we last spoke, I had only succeeded in installing mxfold2 and testing the training script on my local machine and when I had tried running the program on Farnam, only received error messages. I had felt confident that I would be able to install mxfold2 on Farnam, but so far, I haven't been able to succeed.

I tried using pip3 to install both the tarball available on Github and the Linux wheel. Both generated error the following error messages:

ERROR: mxfold2-0.1.1-cp38-cp38-linux_x86_64.whl is not a supported wheel on this platform.

I then decided to install a virtual environment and repeated the same steps without any luck. Next, I tried downloading a Singularity container built for PyTorch, and again got the same error message. I then tried a Singularity container with Python 3.9 installed, and got a bit further, although the installation script output an error message saying that PyTorch was a dependency that had yet to be installed. I then tried installing PyTorch using pip, but received an error message about installation tools. I tried installing these installation tools, which included PEP517, without success. I finally tried building PyTorch from its source wheel, but got similar error messages about the wheels being unsupported on the platform.

Do you know how I might fix this issue?

kad-ecoli commented 3 years ago

Firstly, on installing mxfold2, "is not a supported wheel on this platform" is usually caused by two reasons:

  1. The operating system is not compatible, e.g. install a Mac wheel on Linux.
  2. The python version is not compatible, e.g. install a python3.7 wheel into python3.9

Reason 1 is unlikely, because the "linux_x86_64" wheel is for 64bit Linux, which is the operating system of Farnam. I suspect the reason is reason 2. To check this, you can unzip the wheel, and locate the file *-dist-info/METADATA There, you will see the dependency (Python 3.7, Python3.8). This means that the wheel is not built for python3.9. To address this issue, you have two choices.

  1. Install Python3.7 or Python3.8 from https://repo.anaconda.com/miniconda/ (Miniconda3-py38_4.8.3-Linux-x86_64.sh is for python3.8; Miniconda3-py37_4.8.3-Linux-x86_64.sh is for python3.7).
  2. Recompile mxfold2 from source code. First, git clone https://github.com/keio-bioinformatics/mxfold2. Also download https://github.com/keio-bioinformatics/mxfold2/releases/download/v0.1.1/mxfold2-0.1.1.tar.gz and extract PKG-INFO and setup.py into the cloned mxfold2 folder. You can then recompile and install mxfold2 using python3 setup.py build and python3 setup.py install

For installation of any dependency, including pytorch, you should make sure your package version is neither too old nor too new.

Secondly, for training, your parameter is too rough. You should download https://github.com/keio-bioinformatics/mxfold2/releases/download/v0.1.0/models-0.1.0.tar.gz and check the *.conf files for the detail training hyperparameters. For example, TR0-canonical.conf includes training hyperparameter for the SPOT-RNA pretraining dataset (bpRNA):

--max-helix-length 30 --embed-size 64 --num-filters 64 --num-filters 64 --num-filters 64 --num-filters 64 --num-filters 64 --num-filters 64 --num-filters 64 --num-filters 64 --filter-size 5 --filter-size 3 --filter-size 5 --filter-size 3 --filter-size 5 --filter-size 3 --filter-size 5 --filter-size 3 --pool-size 1 --dilation 0 --num-lstm-layers 2 --num-lstm-units 32 --num-transformer-layers 0 --num-transformer-hidden-units 2048 --num-transformer-att 8 --num-hidden-units 32 --num-paired-filters 64 --num-paired-filters 64 --num-paired-filters 64 --num-paired-filters 64 --num-paired-filters 64 --num-paired-filters 64 --num-paired-filters 64 --num-paired-filters 64 --paired-filter-size 5 --paired-filter-size 3 --paired-filter-size 5 --paired-filter-size 3 --paired-filter-size 5 --paired-filter-size 3 --paired-filter-size 5 --paired-filter-size 3 --dropout-rate 0.5 --fc-dropout-rate 0.5 --num-att 8 --pair-join cat --model MixC --param TR0-canonicals.pth

Also, by default, mxfold2 train only trains 10 epochs, which is often not enough.