Closed MikeKonrad closed 1 year ago
That would be great! Thank you, Kevin!!
Mike
From: Kevin Bui notifications@github.com Sent: Saturday, October 3, 2020 11:19 AM To: bd2kccd/causal-cmd causal-cmd@noreply.github.com Cc: Michael D Konrad mdk@sei.cmu.edu; Mention mention@noreply.github.com Subject: Re: [bd2kccd/causal-cmd] Is Bootstrap implemented for causal-cmd? Not running for FGES (#57)
@MikeKonradhttps://github.com/MikeKonrad It should be. I doesn't seem like it's working at the moment. I will give you an update on when that's fixed. Thanks.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/bd2kccd/causal-cmd/issues/57#issuecomment-703119334, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACI46PUJCNLE2IK4N7VZGV3SI46FNANCNFSM4RSEJPNA.
@MikeKonrad I am going through the old issues and realized that I haven't response back to you. Sorry about that. We have a new release 1.2.1. This should fix some of the bugs. I apologize for getting back to you extremely late.
Hi Kevin, that is super news!
I’ll have to try bootstrap again. In fact, I just ran the bootstrap from the GUI (with FGES) just 5 minutes ago.
Mike
From: Kevin Bui notifications@github.com Sent: Tuesday, January 19, 2021 2:01 PM To: bd2kccd/causal-cmd causal-cmd@noreply.github.com Cc: Michael D Konrad mdk@sei.cmu.edu; Mention mention@noreply.github.com Subject: Re: [bd2kccd/causal-cmd] Is Bootstrap implemented for causal-cmd? Not running for FGES (#57)
@MikeKonradhttps://github.com/MikeKonrad I am going through the old issues and realized that I haven't response back to you. Sorry about that. We have a new release 1.2.1. This should fix some of the bugs. I apologize for getting back to you extremely late.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/bd2kccd/causal-cmd/issues/57#issuecomment-763054662, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACI46PX6WFPSU2CTQP4ZNGTS2XJF5ANCNFSM4RSEJPNA.
@MikeKonrad I haven't tried out bootstrapping yet. So, does it seem to be working now?
Kevin, I’m sorry—there is a task I’m working on right now that requires all my attention through the end of the month, but I will test out bootstrapping using causal-cmd the first week in February. That’s when I will again make use of it.
Mike
From: Kevin Bui notifications@github.com Sent: Tuesday, January 19, 2021 3:54 PM To: bd2kccd/causal-cmd causal-cmd@noreply.github.com Cc: Michael D Konrad mdk@sei.cmu.edu; Mention mention@noreply.github.com Subject: Re: [bd2kccd/causal-cmd] Is Bootstrap implemented for causal-cmd? Not running for FGES (#57)
@MikeKonradhttps://github.com/MikeKonrad I haven't tried out bootstrapping yet. So, does it seem to be working now?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/bd2kccd/causal-cmd/issues/57#issuecomment-763133111, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACI46PQLTFJDNPV5A2VOMX3S2XWNVANCNFSM4RSEJPNA.
@MikeKonrad Sure. No hurry. I was just curious. Thanks.
@MikeKonrad Just to keep you up-to-date. The version 1.2.1 still has issue with missing bootstrapping parameters. I have made a new release of version 1.2.2 that fixes this issue. Please checkout the latest version. Also, I have updated tetrad-lib dependency to version 6.8.1 that fixes the issue with knowledge. Some of the parameter names has changed. For an example, "structurePrior" is now "semBicStructurePrior."
Thank you, Kevin!
I am happy to report causal-cmd does seem to do bootstrapping fine now.
I plan to do a bit more testing over the next few days (e.g., comparing the probability of no edge column from the edge and edge type probability table obtained from the GUI against the same obtained from causal-cmd’s report) before using this feature more extensively the next few weeks. But just looking at the top entries from both tables looks reassuring!
I am very relieved!!
I will let you know what further I learn in a few more days but wanted to reach out to you.
(The webpage, by the way, shows the old form for search parameters, for example sem-bic instead of sem-bic-score. Now someone who carefully reads the documentation will catch this, but I just wanted you to be aware. Shouldn’t be a real high priority, but would be good to fix eventually, especially if you’re going to have newcomers trying to use causal-cmd.)
Thanks, Mike
From: Kevin Bui @.> Sent: Monday, February 22, 2021 12:23 AM To: bd2kccd/causal-cmd @.> Cc: Michael D Konrad @.>; Mention @.> Subject: Re: [bd2kccd/causal-cmd] Is Bootstrap implemented for causal-cmd? Not running for FGES (#57)
@MikeKonradhttps://github.com/MikeKonrad Just to keep you up-to-date. The version 1.2.1 still has issue with missing bootstrapping parameters. I have made a new release of version 1.2.2 that fixes this issue. Please checkout the latest version. Also, I have updated tetrad-lib dependency to version 6.8.1 that fixes the issue with knowledge. Some of the parameter names has changed. For an example, "structurePrior" is now "semBicStructurePrior."
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/bd2kccd/causal-cmd/issues/57#issuecomment-783090704, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACI46PWY5L5IJRO4ZVTWA3DTAHS3DANCNFSM4RSEJPNA.
Kevin and Joe (bringing Joe in because of the question at the end),
I tested causal-cmd bootstrapping pretty thoroughly (by my standards) and other than a slight difference in calibration (what I’ll call it, but I’ll explain more in just a little bit), it seems to perform as accurately/well as does bootstrapping in the Tetrad GUI.
I realize pretty much the same algorithms and code are involved, so that shouldn’t be too surprising.
Not a problem exactly, but I noticed that the probability of no edge values are just a bit higher for causal-cmd use of FGES, SEMBIC, and bootstrapping versus those from using the Tetrad GUI (the table under the resulting search graph has a column titled “No Edge”). I’m not sure why that would be. The two approaches (causal-cmd and GUI) pull from slightly different versions of 6.8.0 baseline I’m betting, so maybe that’s why. And that’s what I meant by slightly different calibration.
Question: Is there any possibility of learning more about the FGES algorithm improvements from recent bug fixing reports? We have a (very) few published or soon-to-be-published papers using FGES and bootstrapping, and that’s why I (deeply) care?
I hope you are all doing well during these strange times.
Thank you, Mike 412-260-5948
From: Michael D Konrad Sent: Sunday, April 11, 2021 10:48 AM To: 'bd2kccd/causal-cmd' @.>; bd2kccd/causal-cmd @.> Cc: Mention @.***> Subject: RE: [bd2kccd/causal-cmd] Is Bootstrap implemented for causal-cmd? Not running for FGES (#57)
Thank you, Kevin!
I am happy to report causal-cmd does seem to do bootstrapping fine now.
I plan to do a bit more testing over the next few days (e.g., comparing the probability of no edge column from the edge and edge type probability table obtained from the GUI against the same obtained from causal-cmd’s report) before using this feature more extensively the next few weeks. But just looking at the top entries from both tables looks reassuring!
I am very relieved!!
I will let you know what further I learn in a few more days but wanted to reach out to you.
(The webpage, by the way, shows the old form for search parameters, for example sem-bic instead of sem-bic-score. Now someone who carefully reads the documentation will catch this, but I just wanted you to be aware. Shouldn’t be a real high priority, but would be good to fix eventually, especially if you’re going to have newcomers trying to use causal-cmd.)
Thanks, Mike
From: Kevin Bui @.**@.>> Sent: Monday, February 22, 2021 12:23 AM To: bd2kccd/causal-cmd @.**@.>> Cc: Michael D Konrad @.**@.>>; Mention @.**@.>> Subject: Re: [bd2kccd/causal-cmd] Is Bootstrap implemented for causal-cmd? Not running for FGES (#57)
@MikeKonradhttps://github.com/MikeKonrad Just to keep you up-to-date. The version 1.2.1 still has issue with missing bootstrapping parameters. I have made a new release of version 1.2.2 that fixes this issue. Please checkout the latest version. Also, I have updated tetrad-lib dependency to version 6.8.1 that fixes the issue with knowledge. Some of the parameter names has changed. For an example, "structurePrior" is now "semBicStructurePrior."
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/bd2kccd/causal-cmd/issues/57#issuecomment-783090704, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACI46PWY5L5IJRO4ZVTWA3DTAHS3DANCNFSM4RSEJPNA.
@jdramsey Hi Joe. I am tagging you just in case you didn't see the message.
Hi Joe and Kevin,
I promised Joe feedback on the latest FGES SEM BIC algorithm changes.
Over the past month, I tested the mid-April release of causal-cmd (let’s call it Apr.cmd for short) against the mid-February release (Feb.cmd), using a few dozen small real-world datasets (e.g., a typical size is 11-13 cases and 30-40 variables).
The bottom-line: all went very well, but there are two issues that I’d like to raise for future consideration:
It’s these lines that I find problematic as there can be so many per search of a bootstrap sample
* Is there any simple way to dispense of all the output of type b?? I like to direct output-to-console to a file for later examination, so just running it is not the best option.
Outline of how I did the testing (read only if you’re interested):
As I’ve said, my datasets are typically rather small, and collected from an electronic/mechanical system, and the causal system is pretty discoverable in such circumstances, though a bit vulnerable to persistent bias in the data or momentary reduction in variable variance (ouch).
For each dataset to be tested against both versions of the algorithm, I do this:
Pre-search, I inject/append null variables into/to the dataset. (Independently and pseudo-randomly, I permute a copy of each variable and add that permutation to the dataset [what Joe was calling a “null variable”])
Post-search, are the resulting PNE columns comparable? I answer this question two ways: (1) Including the null variables, what’s the correlation between the two PNE vectors? (2) Calculate the 5th percentile (say) of adjacencies terminating in at least one null variable. Then only include adjacencies among the original variables whose PNE is < that percentile for just those adjacencies terminating on at least one null variable. Then what’s the size of the symmetric difference of the two trimmed bootstrapped search results (one from each algorithm)? Furthermore, for each adjacency in that symmetric difference, what is its percentile in the algorithm that scored a “miss?” Is it also very low or not?
On both fronts, the Feb.cmd tested very comparably to last year’s GUI (version 20200828); and the Apr.cmd tested very comparably to Feb.cmd. And this was the case for a range of different datasets, though all of them relatively small.
Please reply if you have any questions or comments.
I hope you are both doing well!!!
Thank you, Mike
P.S. Kevin: for an email address, I just grabbed a link associated with a bug report to the github site. I hope that’s okay.
@MikeKonrad I'm sure many of your bootstrap issues will be completely fixed in the next release of Tetrad. Kevin and I (mostly Kevin!) put a lot of work in bootstrapping in the last month or so, and it appears to us to be working perfectly now. But I am unsure about the verbosity of output issue, so if you have any further input about that it would be helpful. The calibration issue is I think completely solved. Sorry for the delay; very small team working on Tetrad as you know!
@MikeKonrad Actually, to be completely honest, I think the verbosity issue has been completely resolved by Kevin's recent efforts in bootstrapping in the forthcoming release,
https://github.com/cmu-phil/tetrad/wiki/Forthcoming-fixes-for-7.1.3_2
but you'll have to tell us. There are the verbose outputs for each algorithm run followed by the verbose output for the bootstrapping report, as below. (By the way, all inaccuracies in the bootstrapping report are now resolved so far as we know.) So you get an output like this:
SO: I guess the question is whether you want to elide the outputs for the individual algorithms runs and just get the bootstrapping report? We'd have to add an extra parameter to accomplish that but it's do-able.
Joe
new thread Bootstrappiung: Number of searches that didn't return a graph = 0 Resampling on the GFCI (Greedy Fast Causal Inference) using Fisher Z test and EBIC Score Running Resamplings in Sequential Mode, numberResampling = 100 new thread thread started ... knowledge being set ... Elapsed initializeForwardEdgesFromEmptyGraph = 0 ms
Graph Edges:
Graph Node Attributes: Score: [X1: 112.26869383021791;X2: 80.34029121893114;X3: -23.746578090494808;X4: 112.64780480901658;X5: -101.14981714038602;X6: -25.30877550381834;X7: 39.372281199422375;X8: 56.82126448164234;X9: -79.57556235320841;X10: 46.642003415041884]
probDistribution finished in 2 ms
I don't know, that is pretty verbose, let me think...
Perhaps, @kvb2univpitt we could set verbose to false for the individual algorithm runs?
@MikeKonrad I think bootstrapping is working much better now--try the most recent version of Tetrad. Actually in a week (Monday?) we'll release another version with what we think if the final bootstrapping fix, but you can be the judge.
Anyway this is an old issue at this point, so I'll close it, but if you have more bootstrapping issues you know where to find us :)
Thank you, sounds good, Joe!
From: Joseph Ramsey @.> Reply-To: bd2kccd/causal-cmd @.> Date: Tuesday, March 14, 2023 at 7:01 PM To: bd2kccd/causal-cmd @.> Cc: Michael Konrad @.>, Mention @.***> Subject: Re: [bd2kccd/causal-cmd] Is Bootstrap implemented for causal-cmd? Not running for FGES (#57)
@MikeKonradhttps://github.com/MikeKonrad I think bootstrapping is working much better now--try the most recent version of Tetrad. Actually in a week (Monday?) we'll release another version with what we think if the final bootstrapping fix, but you can be the judge.
Anyway this is an old issue at this point, so I'll close it, but if you have more bootstrapping issues you know where to find us :)
— Reply to this email directly, view it on GitHubhttps://github.com/bd2kccd/causal-cmd/issues/57#issuecomment-1468979897, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACI46PWDTQAHB3XR7LRIUNDW4D2DJANCNFSM4RSEJPNA. You are receiving this because you were mentioned.Message ID: @.***>
@MikeKonrad It should be. I doesn't seem like it's working at the moment. I will give you an update on when that's fixed. Thanks.