search

ronkeizer / vpc

R library to create visual predictive checks (VPC)
Other
36 stars 20 forks source link

ULOQ and LLOQ behavior for continuous VPCs #43

Closed bguiastr closed 7 years ago

bguiastr commented 7 years ago

I would like to debate over the way LLOQ (and ULOQ) are implemented for continuous VPCs.

Here is an example of VPC for continuous data without LLOQ: vpc_base

Now if a LLOQ is defined in vpc(obs, sim, lloq = 20) all data below LLOQ is like cut from the plot as shown below: vpc_lloq_cut

I believe that it would be more informative to have the following instead: vpc_lloq_2

Which as in xpose4 could be used along with the vpc_cens(obs, sim, lloq = 20) to ensure proper prediction of the proportion of BQL data: vpc_cens

Let me know your thoughts!

ronkeizer commented 7 years ago

hi Ben,

I'm not so sure. I can see your point in a simulation scenario where you indeed have all the values of the censored observations, then the plot you propose is surely more informative. However, the most common use-case (and what this functionality is meant for) is that you don't have the underlying value of the censored data. Since with the VPC we are trying to see whether our simulated data match up with our observed data, I would say we don't want our stats of the simulation data to incorporate any censored data either? We could add the LOQ line with a label in the plot as you did though, to indicate that there is an LOQ?

R

On Thu, Aug 17, 2017 at 1:43 PM, Benjamin notifications@github.com wrote:

I would like to debate over the way LLOQ (and ULOQ) are implemented for continuous VPCs.

Here is an example of VPC for continuous data without LLOQ: [image: vpc_base] https://user-images.githubusercontent.com/7304883/29432499-3ab023ca-839c-11e7-92bb-1eda387d9229.png

Now if a LLOQ is defined in vpc(obs, sim, lloq = 20) all data below LLOQ is like cut from the plot as shown below: [image: vpc_lloq_cut] https://user-images.githubusercontent.com/7304883/29432516-464d34d4-839c-11e7-950a-83dde928b931.png

I believe that it would be more informative to have the following instead: [image: vpc_lloq_2] https://user-images.githubusercontent.com/7304883/29432711-f42ef6a0-839c-11e7-996f-7a96f5e1a546.png

Which as in xpose4 could be used along with the vpc_cens(obs, sim, lloq = 20) to ensure proper prediction of the proportion of BQL data: [image: vpc_cens] https://user-images.githubusercontent.com/7304883/29432800-38ad7d92-839d-11e7-8619-27a105b8bc8e.png

Let me know your thoughts!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ronkeizer/vpc/issues/43, or mute the thread https://github.com/notifications/unsubscribe-auth/AAO4USJssr6WmcC3VGRxim-FVNkC5q6aks5sZKYLgaJpZM4O6yb9 .

andrewhooker commented 7 years ago

Hi Benjamin,

I think that we do this right in xpose4. You cannot compute the real percentiles when they go below/above the LOQ but you can always compute the simulated values.

Andy

Sent from my iPad

On 18 Aug 2017, at 08:08, Ron Keizer notifications@github.com wrote:

hi Ben,

I'm not so sure. I can see your point in a simulation scenario where you indeed have all the values of the censored observations, then the plot you propose is surely more informative. However, the most common use-case (and what this functionality is meant for) is that you don't have the underlying value of the censored data. Since with the VPC we are trying to see whether our simulated data match up with our observed data, I would say we don't want our stats of the simulation data to incorporate any censored data either? We could add the LOQ line with a label in the plot as you did though, to indicate that there is an LOQ?

R

On Thu, Aug 17, 2017 at 1:43 PM, Benjamin notifications@github.com wrote:

I would like to debate over the way LLOQ (and ULOQ) are implemented for continuous VPCs.

Here is an example of VPC for continuous data without LLOQ: [image: vpc_base] https://user-images.githubusercontent.com/7304883/29432499-3ab023ca-839c-11e7-92bb-1eda387d9229.png

Now if a LLOQ is defined in vpc(obs, sim, lloq = 20) all data below LLOQ is like cut from the plot as shown below: [image: vpc_lloq_cut] https://user-images.githubusercontent.com/7304883/29432516-464d34d4-839c-11e7-950a-83dde928b931.png

I believe that it would be more informative to have the following instead: [image: vpc_lloq_2] https://user-images.githubusercontent.com/7304883/29432711-f42ef6a0-839c-11e7-996f-7a96f5e1a546.png

Which as in xpose4 could be used along with the vpc_cens(obs, sim, lloq = 20) to ensure proper prediction of the proportion of BQL data: [image: vpc_cens] https://user-images.githubusercontent.com/7304883/29432800-38ad7d92-839d-11e7-8619-27a105b8bc8e.png

Let me know your thoughts!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ronkeizer/vpc/issues/43, or mute the thread https://github.com/notifications/unsubscribe-auth/AAO4USJssr6WmcC3VGRxim-FVNkC5q6aks5sZKYLgaJpZM4O6yb9 .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

andrewhooker commented 7 years ago

I think like this makes sense.

On 18 Aug 2017, at 8:32 , Andrew Hooker andrew.c.hooker@gmail.com wrote:

Hi Benjamin,

I think that we do this right in xpose4. You cannot compute the real percentiles when they go below/above the LOQ but you can always compute the simulated values.

Andy

Sent from my iPad

On 18 Aug 2017, at 08:08, Ron Keizer <notifications@github.com mailto:notifications@github.com> wrote:

hi Ben,

I'm not so sure. I can see your point in a simulation scenario where you indeed have all the values of the censored observations, then the plot you propose is surely more informative. However, the most common use-case (and what this functionality is meant for) is that you don't have the underlying value of the censored data. Since with the VPC we are trying to see whether our simulated data match up with our observed data, I would say we don't want our stats of the simulation data to incorporate any censored data either? We could add the LOQ line with a label in the plot as you did though, to indicate that there is an LOQ?

R

On Thu, Aug 17, 2017 at 1:43 PM, Benjamin <notifications@github.com mailto:notifications@github.com> wrote:

I would like to debate over the way LLOQ (and ULOQ) are implemented for continuous VPCs.

Here is an example of VPC for continuous data without LLOQ: [image: vpc_base] <https://user-images.githubusercontent.com/7304883/29432499-3ab023ca-839c-11e7-92bb-1eda387d9229.png https://user-images.githubusercontent.com/7304883/29432499-3ab023ca-839c-11e7-92bb-1eda387d9229.png>

Now if a LLOQ is defined in vpc(obs, sim, lloq = 20) all data below LLOQ is like cut from the plot as shown below: [image: vpc_lloq_cut] <https://user-images.githubusercontent.com/7304883/29432516-464d34d4-839c-11e7-950a-83dde928b931.png https://user-images.githubusercontent.com/7304883/29432516-464d34d4-839c-11e7-950a-83dde928b931.png>

I believe that it would be more informative to have the following instead: [image: vpc_lloq_2] <https://user-images.githubusercontent.com/7304883/29432711-f42ef6a0-839c-11e7-996f-7a96f5e1a546.png https://user-images.githubusercontent.com/7304883/29432711-f42ef6a0-839c-11e7-996f-7a96f5e1a546.png>

Which as in xpose4 could be used along with the vpc_cens(obs, sim, lloq = 20) to ensure proper prediction of the proportion of BQL data: [image: vpc_cens] <https://user-images.githubusercontent.com/7304883/29432800-38ad7d92-839d-11e7-8619-27a105b8bc8e.png https://user-images.githubusercontent.com/7304883/29432800-38ad7d92-839d-11e7-8619-27a105b8bc8e.png>

Let me know your thoughts!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/ronkeizer/vpc/issues/43 https://github.com/ronkeizer/vpc/issues/43>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAO4USJssr6WmcC3VGRxim-FVNkC5q6aks5sZKYLgaJpZM4O6yb9 https://github.com/notifications/unsubscribe-auth/AAO4USJssr6WmcC3VGRxim-FVNkC5q6aks5sZKYLgaJpZM4O6yb9> .

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ronkeizer/vpc/issues/43#issuecomment-323268097, or mute the thread https://github.com/notifications/unsubscribe-auth/AFLmZHWpsieX8ed_myVLwtGqGlXb4q6Dks5sZSpTgaJpZM4O6yb9.

bguiastr commented 7 years ago

@ronkeizer I may have gotten a bit confused throughout my explanation, the point I was trying to make is to have it as in xpose4. I would recommend the following for continuous VPCs:

  1. As mentioned by @andrewhooker only the simulations should be shown below the LLOQ (or above the ULOQ).
  2. I would use NA values and not LLOQ for the lines of the observation percentiles, which to me are misleading if they are flat at LLOQ.
  3. Add an horizontal line at LLOQ on the plot

Here is an example taken from M. Bergstrand et al. 2009 (Handling Data Below the Limit of Quantification in Mixed Effect Models)

example

ronkeizer commented 7 years ago

Ok, understand now and makes sense. My thinking got a bit sidetrack by the plot you showed with the obs-line going below loq too. Let me check how xpose 4 is showing this exactly and get back to you.

From: Benjamin notifications@github.com Sent: Friday, August 18, 2017 1:47:34 AM To: ronkeizer/vpc Cc: Ron Keizer; Mention Subject: Re: [ronkeizer/vpc] ULOQ and LLOQ behavior for continuous VPCs (#43)

@ronkeizerhttps://github.com/ronkeizer I may have gotten a bit confused throughout my explanation, the point I was trying to make is to have it as in xpose4. I would recommend the following for continuous VPCs:

  1. As mentioned by @andrewhookerhttps://github.com/andrewhooker only the simulations should be shown below the LLOQ (or above the ULOQ).
  2. I would use NA values and not LLOQ for the lines of the observation percentiles, which to me are misleading if they are flat at LLOQ.
  3. Add an horizontal line at LLOQ on the plot

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ronkeizer/vpc/issues/43#issuecomment-323297396, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAO4Ud190kOKmBI7OlwoMSJNLOwSWeljks5sZU-mgaJpZM4O6yb9.

andrewhooker commented 7 years ago

I can look into this if you like.

On 18 Aug 2017, at 16:17 , Ron Keizer notifications@github.com wrote:

Ok, understand now and makes sense. My thinking got a bit sidetrack by the plot you showed with the obs-line going below loq too. Let me check how xpose 4 is showing this exactly and get back to you.

From: Benjamin notifications@github.com Sent: Friday, August 18, 2017 1:47:34 AM To: ronkeizer/vpc Cc: Ron Keizer; Mention Subject: Re: [ronkeizer/vpc] ULOQ and LLOQ behavior for continuous VPCs (#43)

@ronkeizerhttps://github.com/ronkeizer I may have gotten a bit confused throughout my explanation, the point I was trying to make is to have it as in xpose4. I would recommend the following for continuous VPCs:

  1. As mentioned by @andrewhookerhttps://github.com/andrewhooker only the simulations should be shown below the LLOQ (or above the ULOQ).
  2. I would use NA values and not LLOQ for the lines of the observation percentiles, which to me are misleading if they are flat at LLOQ.
  3. Add an horizontal line at LLOQ on the plot

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ronkeizer/vpc/issues/43#issuecomment-323297396, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAO4Ud190kOKmBI7OlwoMSJNLOwSWeljks5sZU-mgaJpZM4O6yb9. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ronkeizer/vpc/issues/43#issuecomment-323365918, or mute the thread https://github.com/notifications/unsubscribe-auth/AFLmZMPCwEGAO4MaBT7W_5C9uhvHvehVks5sZZz5gaJpZM4O6yb9.

ronkeizer commented 7 years ago

Sure, sounds good. It shouldn't be a large change in my code, I think.

On Fri, Aug 18, 2017 at 8:02 AM, Andrew Hooker notifications@github.com wrote:

I can look into this if you like.

On 18 Aug 2017, at 16:17 , Ron Keizer notifications@github.com wrote:

Ok, understand now and makes sense. My thinking got a bit sidetrack by the plot you showed with the obs-line going below loq too. Let me check how xpose 4 is showing this exactly and get back to you.

From: Benjamin notifications@github.com Sent: Friday, August 18, 2017 1:47:34 AM To: ronkeizer/vpc Cc: Ron Keizer; Mention Subject: Re: [ronkeizer/vpc] ULOQ and LLOQ behavior for continuous VPCs (#43)

@ronkeizerhttps://github.com/ronkeizer I may have gotten a bit confused throughout my explanation, the point I was trying to make is to have it as in xpose4. I would recommend the following for continuous VPCs:

  1. As mentioned by @andrewhookerhttps://github.com/andrewhooker only the simulations should be shown below the LLOQ (or above the ULOQ).
  2. I would use NA values and not LLOQ for the lines of the observation percentiles, which to me are misleading if they are flat at LLOQ.
  3. Add an horizontal line at LLOQ on the plot

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ ronkeizer/vpc/issues/43#issuecomment-323297396, or mute the thread< https://github.com/notifications/unsubscribe-auth/ AAO4Ud190kOKmBI7OlwoMSJNLOwSWeljks5sZU-mgaJpZM4O6yb9>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/ronkeizer/vpc/issues/43#issuecomment-323365918>, or mute the thread https://github.com/notifications/unsubscribe- auth/AFLmZMPCwEGAO4MaBT7W_5C9uhvHvehVks5sZZz5gaJpZM4O6yb9.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ronkeizer/vpc/issues/43#issuecomment-323378029, or mute the thread https://github.com/notifications/unsubscribe-auth/AAO4Ufvvp_hneOn9BtPbMkMrsJFxwQt7ks5sZaeRgaJpZM4O6yb9 .