Background fitting

[DanielisRutkauskas]

Executing

d.calc_bg(bg.exp_fit, time_s=3600, tail_min_us=4000) dplot(d, hist_bg, show_fit=True)

produces the attached picture

where the approximation is obviously off the experimental curves. What is the reason for this and how could it be corrected?

[tritemio]

This issue is only a visualization one. If you notice, the slope of the exponential seems to be fitted well, only the normalization is wrong. I recently had a similar issue when using short time periods (1s) for background estimation. I committed a fix to master (ddede3025fdf28ffa676cd250121ba6c95f76907) that will hopefully fix also your case.

The reason of the issue is the following. The background fitting does not fit the histogram but is a MLE estimator of the rate. The "amplitude" of the exponential (i.e. the normalization of the model) is computed by the plotting function itself for visualization purposes only. To compute the normalization constant we need to fix a threshold above which the histogram is considered exponential and normalize the model to this tail. Apparently, the previous logic (that used threshold = 1/4 * 1/ bg_rate) was not working well is some cases. The new approach uses threshold = max_non_zero_bin / 3 and works well in all the conditions I tested.

Can you try current master on your file to see if the issue is fixed? I can give instructions if needed.

[DanielisRutkauskas]

So basically computation is correct, visualization is not completetly accurate. Indeed, I would need instructions on how to use the current master.

[tritemio]

Do you have git? If yes, download the FRETBursts repo with:

git clone https://github.com/tritemio/FRETBursts.git

If you don't have git, download the FRETBursts ZIP archive and decompress it in some folder.

In both cases you need to enter the FRETBursts source folder from the terminal using cd. Then install fretbursts from the current folder with:

pip install -e .

Now, restart a notebook. When FRETBursts is imported it should print the version like this:

You are running FRETBursts (version 0.6.3+78.gfb59a38).

It means you are using a version which as 78 commits after 0.6.3 and the last commit is fb59a38.

To go back to the stable FRETBursts open the terminal (any folder) and type:

pip uninstall fretbursts

and

conda install fretbursts=0.6.3

[tritemio]

Thinking about it, if you download the ZIP without git, you will not get the exact version correctly printed because the information is retrieved through git. The other steps should work though. The bonus of installing git is that it adds a context menu (right click) to file explorer on windows to easily open a terminal in a specific folder (git bash menu), this will be more convenient than using cd to change folder.

[DanielisRutkauskas]

So installing the suggested fb59a38 is only for temporary use? I would have to go back to 0.6.3 in the end?

[tritemio]

You don't have to, unless you find a bug that prevents you from using FRETBursts. The master branch (which is the main branch on github), it is just a little less tested. As a rule, all new development takes place on separate branches, and I strive to keep the master branch always stable and releasable. The advantage of installing from master is that you can easily get updates right away without waiting for a formal release. In case you find some errors you can always roll back to the latest released version to get your job done. Since you have the full version down to the commit level printed in the notebook you will know which version works and which it does not.

Another even safer approach is creating a separate environment with conda. You can create a new environment in which you install a totally different set of packages. You can have an environment with the latest released FRETBursts and one with the latest master version.

The default environment is called root. To make a new environment called fbmaster:

conda create -n fbmaster python=3.6 fretbursts

Then activate the environment (windows version):

activate fbmaster

Once the environment is activated you can install/remove more packages in it. For example you can replace the stable FRETBursts with the version from github master using pip install -e . in the same terminal where the environment has been activated. Installing the stable FRETBursts first, allows installing all the dependencies through conda. Conda adds all the environment to the notebook menu. So when you open a notebook, you can go to the menu Kernel -> Change kernel and select fbmaster instead of default (or vice versa). The latest used kernel is saved in the notebook so you don't have to switch every time.

Environments help to be more reproducible in computations. They can be "saved" or exported to a text file so that they can be recreated on a different machine. For example, you can have an environment for a paper. 1 or 2 years later, an updated python installation may have some incompatibilities causing the old analysis to fail or to give different results. If you saved the environment file, you can restore the old environment with the exact version of all packages with one command. Having a saved environment saves you the trouble of finding and fixing these incompatibilities (they are almost always trivial but it takes time).

[DanielisRutkauskas]

Thank you for this extensive explanation. I will try to follow these recommendations.

[tritemio]

FYI, @mayeshh sent a PR to add how-to install FRETBursts in a separated environment to the documentation. Here it is:

http://fretbursts.readthedocs.io/en/latest/installation.html#install-fretbursts-in-a-separate-environment

[tritemio]

@DanielisRutkauskas, FRETBursts 0.6.4 is released and includes the fix (see last point in the release notes):

• release notes