kvb2univpitt
commented
5 years ago @azzaea Did you create a application-h2.properties like this one https://github.com/bd2kccd/causal-web/issues/78#issuecomment-523624927 ?
Open azzaea opened 5 years ago
kvb2univpitt
commented
5 years ago @azzaea Did you create a application-h2.properties like this one https://github.com/bd2kccd/causal-web/issues/78#issuecomment-523624927 ?
azzaea
commented
5 years ago Yes, I copied the exact same file application-h2.properties over, modified the application.properties entry: spring.profiles.active=scheduler,h2 and added the same dependency lines (repeated below) to the pom.xml file
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>1.4.199</version>
</dependency>
@azzaea Sorry for the late reply. I have been busy with other projects.
By the way, you don't need the version tag. H2 is already part of Spring Boot. You can just add the following to the POM.xml:
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
</dependency>Remove the following line from the JPA section in the application.properties file:
spring.jpa.properties.hibernate.globally_quoted_identifiers=trueThe application.properties file should look like this:
# APPLICATION SETTINGS (SpringApplication)
spring.main.banner-mode=off
# PROFILES
# scheduler, slurm
# hsqldb, mysql
spring.profiles.active=scheduler,h2
# LOGGING
logging.file=causal_rest_api.log
logging.level.*=INFO
logging.level.org.springframework.web=INFO
logging.level.org.hibernate=ERROR
# EMBEDDED SERVER CONFIGURATION (ServerProperties)
server.port=9000
server.session-timeout=-1
server.context-path=/ccd-api/
# JPA (JpaBaseConfiguration, HibernateJpaAutoConfiguration)
# Hibernate ddl auto (create, create-drop, update)
spring.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=false
# MULTIPART (MultipartProperties)
# Enable support of multi-part uploads
spring.http.multipart.enabled=true
spring.http.multipart.max-file-size=-1
spring.http.multipart.max-request-size=-1Thank you much for your patience @kvb2univpitt !
The API server now works- the last few lines in the log are below. However, I'm not able to move from this step forward. I understand this sets the server at: 0.0.0.0:9000 (which is what I should use instead of the Pittsburgh supercomputer server https://ccd4.vm.bridges.psc.edu demonstrated in the documentation here).
As a starter, the GET command returns back authentication error, and the POST doesn't give anything. Would you kindly advise?
Time and again, very grateful for you help indeed. Azza
$ GET 0.0.0.0:9000/ccd-api/jwt
{"timestamp":1567622489648,"status":401,"error":"Unauthorized","message":"User credentials are required.","path":"/jwt"}
$ POST /ccd-api/jwt HTTP/1.1
Please enter content (application/x-www-form-urlencoded) to be POSTed:
Host: 0.0.0.0:9000
Authorization: Basic ZGVtb0BwaXR0LmVkdToxMjM=
# Nothing happens- I have to manually cancel the commandAnd, on a different terminal- the causal-rest-api is on:
$ java -jar causal-rest-api.jar
:
:
019-09-04 20:29:52.055 INFO 21653 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2019-09-04 20:29:52.094 INFO 21653 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 0
2019-09-04 20:29:53.563 INFO 21653 --- [ main] o.e.j.s.h.ContextHandler.application : Initializing Spring FrameworkServlet 'dispatcherServlet'
2019-09-04 20:29:53.565 INFO 21653 --- [ main] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization started
2019-09-04 20:29:53.607 INFO 21653 --- [ main] o.s.web.servlet.DispatcherServlet : FrameworkServlet 'dispatcherServlet': initialization completed in 42 ms
2019-09-04 20:29:53.656 INFO 21653 --- [ main] o.e.jetty.server.AbstractConnector : Started ServerConnector@2cdd0d4b{HTTP/1.1,[http/1.1]}{0.0.0.0:9000}
2019-09-04 20:29:53.659 INFO 21653 --- [ main] .s.b.c.e.j.JettyEmbeddedServletContainer : Jetty started on port(s) 9000 (http/1.1)
2019-09-04 20:29:53.667 INFO 21653 --- [ main] e.p.d.c.c.r.a.CausalRestApiApplication : Started CausalRestApiApplication in 18.04 seconds (JVM running for 18.888)
@azzaea You will need to create a user account in the database first. Use causal-web to create a user account. Causal web and causal rest api should be connecting to the same database.
azzaea
commented
5 years ago Glad to hear from you @kvb2univpitt ! Thank you! Yes, I'm connecting to the same database, but not sure how to feed the data. I'm using my email and 123456 password, and at loss as to proceeding further
azzaea
commented
5 years ago 
I tried feeding the username from this window, or using the email instead, but neither worked
kvb2univpitt
commented
5 years ago @azzaea Sorry for the delay. Let me set this up on my local computer. I'll get back to you on this.
kvb2univpitt
commented
5 years ago First you need to the jwt token using basic authentication, as mentioned here : Getting JSON Web Token(JWT)
Below is the screenshot of how you would get the jwt token:
Note that you need the userId and the jwt token to make other REST calls.
Let's say you want to retrieve a list of dataset, List all dataset files of a user. Below is the screenshot on how you would make the call:
Note that your userId, in this particular example, is 1. Based on this example, you must add the Authorization header with value Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJodHRwOi8vbG9jYWxob3N0LyIsInVpZCI6MSwiZXhwIjoxNTY4MTM1NTM5NjU0LCJpYXQiOjE1NjgxMzE5Mzk2NTR9.JVI-U2BHd1IFoOY0WrPOX0fIy_2KHAubvHCJKMhTM7M.
azzaea
commented
5 years ago Thank you. Appreciated.
I apologize in advance if it seems stupid, but what interface provide these snapshots? I can't access similar content from the causal-web interface, and neither putting the GET command on the terminal directly
kvb2univpitt
commented
5 years ago This is a REST plugin for Firefox: https://github.com/odin-public/RESTED-APS Just search for REST client plugin in Firefox and you will see it.
kvb2univpitt
commented
5 years ago By the way, if you don't mind me asking, what are you using causal-web and causal-rest-api for? I am just curious.
azzaea
commented
5 years ago That's handy- thanks! It still give the same error though:
The database server is up, causal-web is up and causal-rest-api too (the 3 panels in this terminal window)
I'm working with a large dataset (snp data/GWAS), and interested in exploring the fGES-MB method in that setting. Results from the paper look promising, so trying to replicate for a starter at that scale
kvb2univpitt
commented
5 years ago If you're working with large dataset, I suggest you use causal-cmd, a command-line version of Tetrad search algorithms. The documentation for it is here. With causal-cmd you can run fGES-MB on a High Performance Computing (HPC) cluster.
If you're using H2, an in-memory database, you will lose everything in the database if you shut down the service. If you shutdown and start your services again, you will need to recreate your user account. Also, H2 database can only accept one connection at a time. So, if you run causal-web with H2 and causal-rest-api with H2, you're basically running two different instances of H2 database, each can only take one connection. H2 database will not work with this set up. You will need a database with non-volatile storage and can handle multiple connection such as MySQL, SQL Server, MariaDB, etc.
azzaea
commented
5 years ago Akh! Yes, I was using causal-cmd, but understood that I should be using causal-web-api to submit jobs to the cluster as per this comment.
Packaging causal-cmd commands in a usual batch script doesn't quite scale up (fGES-MB took good 10 days on a dataset of 50k feature and 8000 samples; with 10 threads- without actually finishing).
kvb2univpitt
commented
5 years ago Let me ask Dr Ramsey if he has any suggestion on what parameters to use for your dataset.
@jdramsey Do you have any suggestion for running fGES-MB on SNP data with 50k variables and 8k cases?
azzaea
commented
5 years ago Thank you much! This is really appreciated
jdramsey
commented
5 years ago @azzaea @kvb2univpitt The danger is always that the underlying model might be very dense; FGES can bog down a lot with dense graphs. The thing to do is to start with a very high penalty, see if you can get an answer, then if you can, try again with lower penalties until you see what is feasible for you.
azzaea
commented
5 years ago Thank you much @jdramsey A couple of remarks:
I don't see a parameter to control the penalty in causal cmd when used with fges-mbaway from --penaltyDiscount generic parameter, but it is unrecognised:
$ java -jar causal-cmd-1.1.1-jar-with-dependencies.jar --algorithm fges-mb --data-type mixed --dataset charity.txt --delimiter tab --score cg-bic-score --numCategories 3 --targetName Impact --penaltyDiscount 2
Unrecognized option: --penaltyDiscount
usage: java -jar causal-cmd-1.1.1.jar --algorithm fges-mb --data-type mixed --dataset charity.txt --delimiter tab --numCategories 3 --score cg-bic-score --targetName Impact [--choose-dag-in-pattern] [--choose-mag-in-pag] [--comment-marker <string>] [--exclude-var <file>] [--experimental] [--extract-struct-model] [--faithfulnessAssumed] [--generate-complete-graph] [--genereate-pag-from-dag] [--genereate-pag-from-tsdag] [--genereate-pattern-from-dag] [--help] [--help-all] [--json-graph] [--knowledge <file>] [--make-all-edges-undirected] [--make-bidirected-undirected] [--make-undirected-bidirected] [--maxDegree <integer>] [--metadata <file>] [--missing-marker <string>] [--no-header] [--out <directory>] [--prefix <string>] [--quote-char <character>] [--skip-latest] [--skip-validation] [--thread <string>] [--verbose] [--version]
--choose-dag-in-pattern Choose DAG in Pattern graph.
--choose-mag-in-pag Choose MAG in PAG.
--comment-marker <string> Comment marker.
--exclude-var <file> Variables to be excluded from run.
--experimental Show experimental algorithms, tests, and scores.
--extract-struct-model Extract sturct model.
--faithfulnessAssumed Yes if (one edge) faithfulness should be assumed
--generate-complete-graph Generate complete graph.
--genereate-pag-from-dag Generate PAG from DAG.
--genereate-pag-from-tsdag Generate PAG from TsDAG.
--genereate-pattern-from-dag Generate pattern graph from PAG.
--help Show help.
--help-all Show all options and descriptions.
--json-graph Write out graph as json.
--knowledge <file> Prior knowledge file.
--make-all-edges-undirected Make all edges undirected.
--make-bidirected-undirected Make bidirected edges undirected.
--make-undirected-bidirected Make undirected edges bidirected.
--maxDegree <integer> The maximum degree of the graph (min = -1)
--metadata <file> Metadata file. Cannot apply to dataset without header.
--missing-marker <string> Denotes missing value.
--no-header Indicates tabular dataset has no header.
--out <directory> Output directory
--prefix <string> Output file name prefix.
--quote-char <character> Single character denotes quote.
--skip-latest Skip checking for latest software version.
--skip-validation Skip validation.
--thread <string> Number threads.
--verbose Yes if verbose output should be printed or logged
--version Show version.
Use --help for guidance list of options. Use --help-all to show all options.For completion, I have been using synthetic data generated from the distribution below, where I'm interested in the Markov Blanket of y, which is a function of some of the columns of X- hence X_causal:
I'm very very appreciative and open to suggestions, especially that even with smaller dataset (eg 5k features, of which 250 are causal; along 400 samples), the command works, but with rather disappointing results (a 2 node graph, neither of which is from the causal set)
kvb2univpitt
commented
5 years ago @azzaea Please use sem-bic score.
azzaea
commented
5 years ago @kvb2univpitt I thought the Conditional Gaussian score, ( based on this paper ) is the right score to use with mixed data (in my application the predictors are discrete, and the target is continuous), while the sem bic is appropriate for purely continuous data.
Is this not the case?
kvb2univpitt
commented
5 years ago @azzaea You are correct. Sorry. I wasn't being clear. The sem-bic score has penaltyDiscount parameter. That's why you're getting that error.
azzaea
commented
5 years ago Oh- I see. Thank you. But, it wouldn't be applicable to the data I'm working with
Hello again, I'm wondering if
causal-rest-apicould also work withh2instead ofmySQLandHSQL? I'm asking because it was easiest to set uph2while setting up causal-web, while adding a configuration file and a dependency to thepom.xmlfile as in here.With the same settings running
causal-rest-apigenerates the error below:If it is not much to ask, a workaround with h2 would really be appreciated, as it has been quite straight forward to work with it thus far. Else, would you kindly be able to provide a hint into configuring mySQL? I tried to download a server from here, but I see this application (MySQL Workbench) in my PC (shared university PC, so no root access); but I'm not sure where to go
My due gratitudes, and apologies for the volume of these issues.
Kind regards, Azza