search

tdgrant1 / denss

Calculate electron density from a solution scattering profile
GNU General Public License v3.0
31 stars 20 forks source link

Feature request: Real chi^2 values for fit of model to data #27

Closed jbhopkins closed 1 year ago

jbhopkins commented 1 year ago

I'd love to get a real reduced chi^2 value of the fit of the model to the data. The chi^2 reported by DENSS is for the smoothed IFT data binned to the shannon channel points (I believe), and the errors aren't right so the value doesn't have any real meaning (it's only useful in a relative sense for convergence). How hard would it be to calculate the fit profile at the q points of the original experimental data just once, once you've created the final model, and then actually calculate a real reduced chi^2 that could be used to talk about how well the model fits the data?

jbhopkins commented 1 year ago

Another point, this would also allow for a much finer sampling of the residual, which could be much more informative on the fit quality as well.

tdgrant1 commented 1 year ago

Yes, this can be done, and fairly soon. We are about to push DENSS v1.7.0 which includes a tool for fitting experimental scattering profiles from atomic models using density maps, and for that we developed a proper chi2 calculator. I will incorporate this feature into the output _map.fit file from denss. However, it will require a little finagling to feed the experimental data to DENSS, since DENSS does not technically require experimental data currently, but only uses the smoothed IFT fit. However, I can just put some conditionals in there to check for real data.

Tom

On Thu, Jun 1, 2023 at 1:19 PM jbhopkins @.***> wrote:

Another point, this would also allow for a much finer sampling of the residual, which could be much more informative on the fit quality as well.

— Reply to this email directly, view it on GitHub https://github.com/tdgrant1/denss/issues/27#issuecomment-1572478381, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOBJZZHW4GQVDZEVYJGV6DXJDFL5ANCNFSM6AAAAAAYXHI6MU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

jbhopkins commented 1 year ago

Sounds great! Honestly, for my purposes, I just need a fit at the original q spacing. I can make the rest of it work in RAW (chi^2 calculation, residuals, etc) since I have the experimental data there. But if you incorporate it into DENSS itself that's even easier.

tdgrant1 commented 1 year ago

DENSS v1.7.0 has now been pushed to the github repo! This includes a new script called denss.mrc2sas.py which fits a scattering profile calculated from an MRC density map to an experimental dataset, on the qgrid of the experimental data, and calculates a proper chi2 value. So you can take a look there for how to do it yourself in RAW if you like.

This feature is also baked directly into the denss() function, so any internal use of it in RAW should also output the new fits. The denss() function saves the new fit in the _map.fit file, but also returns the fit as a numpy array that can conveniently be used to make your own plots or calculate chi2 yourself for example. I've attached an example plot that denss.py produces with residuals.

Let me know if you run into any errors. We tested it lots, but there's bound to be some bugs.

This version also includes a new class PDB2MRC and corresponding denss.pdb2mrc.py script which calculates highly accurate scattering profiles from pdb files, and the corresponding density maps, and fits to experimental scattering profile. Works like CRYSOL kind of. The publication describing that has been posted to bioRxiv and will be submitted soon.

Tom

On Thu, Jun 1, 2023 at 5:15 PM jbhopkins @.***> wrote:

Sounds great! Honestly, for my purposes, I just need a fit at the original q spacing. I can make the rest of it work in RAW (chi^2 calculation, residuals, etc) since I have the experimental data there. But if you incorporate it into DENSS itself that's even easier.

— Reply to this email directly, view it on GitHub https://github.com/tdgrant1/denss/issues/27#issuecomment-1572795129, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOBJZ4R2FBL63WSP2TGJVDXJEBAFANCNFSM6AAAAAAYXHI6MU . You are receiving this because you commented.Message ID: @.***>

jbhopkins commented 1 year ago

Sounds good. I'm working on a new release for RAW, and I've already updated DENSS for that, so it won't make it into the next version, but I'll incorporate it into the one after it.