Closed GeoffBryan closed 7 years ago
I'm simulating the SED of the object LRLL 31, so the directory containing the simulations on lustre is: LRLL31-puff. There is a separate sub-directory for each simulation as shown in this screen dump:
Inside each of these subdirectories is a parameter file called mctherm.par which contains 149 parameters (mctherm.par.txt) which I need to parse and add to the database.
' =================== Parameter file ==================='
' Preliminaries ' 200000000 = NP = Number of photons from central star 1000000 = NPMIN = Minimum number of photons for Lucy temperature calculation 0 = NIMIN = Minimum number of iterations (Lucy temperature and/or HSEQ) 5 = NIMAX = Maximum number of iterations (Lucy temperature and/or HSEQ) NO = CPEEL = YES for peeling, NO for standard MC YES = CLUCY = YES to use Lucy method for T-corr, NO for Bjorkman & Wood -2211 = I1 = Random number seed 100000 = IWRITE = output timing information every IWRITE photons YES = CWARN = YES to print warnings & errors, NO to not. '../parfiles' = PARDIR = directory containing atmosphere, dust, and filter files NO = DENSINPUT = YES for reading in density grid, NO for calculating analytic
' Central Source Properties ' YES = CLIMB = YES for limb darkening of central source, NO for not YES = CPLANCKST = YES to use planck function for star, 'NO to use stellar atmosphere file kt05750g+3.5z+0.0.ascii = ATNAME = Stellar photosphere file 2.3 = RSTAR = Stellar radius in Solar radii 5.70e3 = TSTAR = Blackbody temperature of central star (K) 1.6 = MASSC = Mass of central star (for TSC properties)
' Outside Illumination ' NO = COUTILLUM = YES for outside illumination (ISRF), NO for none
' Overall Disk Properties '
0.01 = MASSD = Disk mass in solar masses
AU = CRMIN = 'Units of Rmin: Rstar, Rsub, or AU'
RSTAR = CZMIN = 'RSTAR' to set ZMIN at RSTAR, 'RSUB' to scale ZMIN at Rsub, 'R100' to set ZMIN at 100AU
0.2 = FMASSD1 = Fraction of mass in disk 1 (Large grains settled disk)
CYL = CDISKCURV = define disk surfaces/density using spherical radius (SPH) or cylindrical (CYL)
NO = CHSEQ = solve for vertical hydrostatic equilibrium (YES or NO).
1 = ITER_HSEQ = start HSEQ solution after this number of iterations (range 1-NIMAX) (default is 1)
2 = DISK_HSEQ = apply HSEQ to big grain settled disk (1), small grain disk (2), or both (3) (default is 2)
'* GAPs, SPIRALs, and/or WARPs in disk (3-D geometries: set NPG=120 or some other 3-D value) ' NO = CGAPD = gap in disk (YES or NO)
' Large grains settled disk ' www003.par = DUSTNAME(1) = Dust file, thermal grains [disk 1] draine_opac_new.dat = DUSTNAME(5) = Dust file, <200A grains [disk 1] 0.0 = FSG(1) = Fraction of the mass in <200A grains [disk 1] 0.005 = ZSCALE(1) = Scale height in RSTAR (if CZMIN='RSTAR'), HSEQ (if CZMIN='RSUB'), or AU (if CZMIN='R100') [disk 1] 0.13 = RMIND(1) = Minimum disk radius (unit set in CRMIN above) 300.0 = RMAXD(1) = Disk outer radius in AU 2.25 = A(1) = Disk radial density exponent (~ r^(-a)) [disk 1] 1.25 = B(1) = Disk scale height exponent (~ r^(b)) [disk 1] 0.005 = ZSCALE_GAP(1) = Scale height of gap (use same units as ZSCALE) 2.25 = A_GAP(1) = gap radial density exponent 1.25 = B_GAP(1) = gap scale height exponent 0.0001 = RHOSCALE_GAP(1) = ratio of gap surface density just inside RGAPD2 to value just outside
' Small grains disk ' kmh.par = DUSTNAME(2) = Dust file, thermal grains [disk 2] draine_opac_new.dat = DUSTNAME(6) = Dust file, <200A grains [disk 2] 0.0 = FSG(2) = Fraction of the mass in <200A grains [disk 2] 0.01 = ZSCALE(2) = Scale height in RSTAR (if CZMIN='RSTAR'), HSEQ (if CZMIN='RSUB'), or AU (if CZMIN='R100') [disk 2] 0.13 = RMIND(2) = Minimum disk radius (unit set in CRMIN above) 300.0 = RMAXD(2) = Disk outer radius in AU 2.25 = A(2) = Disk density exponent (~ r^(-a)) [disk 2] 1.25 = B(2) = Disk scale height exponent (~ r^(b)) [disk 2] 0.01 = ZSCALE_GAP(2) = Scale height of gap (use same units as ZSCALE) 2.25 = A_GAP(2) = gap density exponent 1.25 = B_GAP(2) = gap scale height exponent 0.0001 = RHOSCALE_GAP(2) = ratio of gap surface density just inside RGAPD2 to value just outside
' Radial Exponential cutoff to disk ' NO = CRADEXP = add radial exponential cutoff to disk density (YES or NO)? 50 = RADEXP = scale for exponential cutoff in AU
' Disk/star accretion properties ' YES = CDISKACC = YES to include disk accretion luminosity, NO to not NO = CALPHA = YES to read alpha disk parameter, NO to read disk Mdot 0.40e-8 = ALPHA/MDOT = Disk alpha parameter (if CALPHA=YES) or accretion rate (if CALPHA=NO) 5.0 = RTRUNC = Magnetosphere co-rotation radius in RSTAR NO = CSPOT = YES if spots on star, NO if not
' Envelope Properties ' r400_ice095.par = DUSTNAME(3) = Dust file, thermal grains [envelope] draine_opac_new.dat = DUSTNAME(7) = Dust file, <200A grains [envelope] 0.0 = FSG(3) = Fraction of the mass in <200A grains [envelope]
' SG properties ' 0.09 = RMIN_SG = minimum radius of <200A grains (units of CRMIN)
' Output Data ' YES = IMFILT = Output images convolved with filters NO = IMCUBE = Output multi-wavelength data cubes (very cool, try this) 149 = NXIMG = number of pixels in one dimension of square image 1 = NPEEL = Number of peel-off angles for SEDs and Images 1 = NAP = Number of apertures for SEDs
' Advanced ' YES = DIFFUSION = Whether to use the diffusion
' Grid parameters ' 400 = NRG = Number of radial cells 197 = NTG = Number of theta cells (use an odd number) (set to 3 for 1-D models). 2 = NPG = Number of phi cells (set to 2 for 1-D or 2-D models).
When the simulation completes one or more fits files are produced as well as a log file I can use the existence of these files (and their contents) to update the records in the database to say that the simulation is: A - not yet run, B completed successfully, C completed but failed.
I have never created an SQL database. May be @Psykar, @amraziz8, @gpoole, @macrocosme know?
I'll try to go through your points here. @amraziz8 may want to clarify some details from a gSTAR standpoint.
If it's small, your machine can be fine. However, you seem to mention that it's growing quite drastically. So perhaps it may be better to start a DB via ITS.
If you know MySQL already, I would say, go for it!
MySQL is available, and I think postgres is available now too.
I would say to go with sqlalchemy.
I would point to their location.
Not impossible! I will let Amr comment on this!
SQLAlchemy. Or, MySQL workbench.
You'll have to get use to the details and set up/conceptualize your database. Once all is set up though, you'll have quick access to all your results and be able to run some nice analytics on your results.
Hope this helps!
From chatting with @GeoffBryan, it seems that the issue is resolved. Closing.
The Problem - too many simulations to keep track of!
I run, or rather have run, lots of (at present ~300) radiative transfer code simulations, using a code called Hochunk on the Swinburne super-computer. I'm doing this to model the SED's of accretion disks around Young Stellar Objects:![inclined-012](https://cloud.githubusercontent.com/assets/20568222/26294625/829f49f0-3f08-11e7-8704-523b7ce4e05f.png)
Here are a few observed Spectral Energy Distributions (SED's):![image](https://cloud.githubusercontent.com/assets/20568222/26294682/e70a3580-3f08-11e7-97cc-13469eafbe56.png)
Currently I keep track of them using an Excel spreadsheet (which is growing unwieldy) so I was thinking it might be useful to replace this with a small SQL database and a number of python scripts.
Advice wanted!
I'm looking for some advice on how to do this (without going overboard) and I thought that it might be a useful thing to talk about at code review as it’s a problem many of us (in a general sense) might come across (and using SQL and python together his likely to be a really useful general skill to pick-up).
A Proposal - SQL database plus Python scripts
Here's my proposal:
add_db_entry.py "this_sub-directory"
This should search for the parameter file, parse it, check to see if it is a new entry in the db and then if it isn’t add a record to the DB and also record whether the output Fits files are in the directory (I.e. The simulation completed successfully) b). Update a record of an existing simulation to the local/remote DB:
update_db_entry.py "this_subdirectory"
(i.e. updates when the simulation is complete along with status) c). Remove an record of an existing simulation on the DB:remove_db_entry.py "this_subdirectory"
d). Query whether an existing parameter file matches any in the DB:query_unique_db.py "mctherm.par"
(Returns NULL if there is no match or returns the names and relative directory paths of those simulations that do match) e). Query db using user's designated SELECT statements and return a report:query_db query_name.qry output.csv
Issues
And here are a few of my issues:
The reasons why.....
One of the reasons for wanting to do this is to stop me running duplicate simulations – they are expensive in time and resources and I don’t want to re-run things I’ve already done – so it makes sense to find a systematic way of determining whether a set of parameters has already been simulated.
Hence if I had a python query "engine" that allowed me to put in ranges of values for various parameters and see if there is a match (or not) in the DB would be useful.
Extra Information
Extra info (a picture of the sub-directory structure, an example of the parameter file) is attached in the next comment from me below.
Cheers, Geoff