Issue phosim catalog sn - Githubissues

LSSTDESC / sims_GCRCatSimInterface

LSST sims interface to gcr-catalogs

BSD 3-Clause "New" or "Revised" License

1 stars 5 forks source link

Issue phosim catalog sn #19

Closed rbiswas4 closed 6 years ago

rbiswas4 commented 6 years ago

Committed changes for writing out phosim catalogs of SN populations.

Inputs

There are data files in data/ that represent SN in protoDC2 hosts in the Main Survey Region and the uDDF region separately. There are also three files that include SN that in hostless in the uDDF region and those in the Main Survey at z < 1.0 and those in the Main Survey at z > 1.0 . These files are:

MainSurveyHostedSNPositions_trimmed.csv
MainSurvey_hostlessSN_trimmed.csv
MainSurvey_hostlessSN_highz_trimmed.csv
uDDFHostedSNPositions_trimmed.csv
uDDF_hostlessSN_trimmed.csv

They include the position and properties of the SN using which catsim can create images.

The catalog classes

These are two new classes in python/desc/sims/GCRCatSimInterface/CatalogClasses.py

SNFileDBObject : which uses catsim functionality to read from the csv files rather than fatboy which we did in Twinkles.
DC2PhosimCatalogSN : Class to modify phosim catalogs for SN for the DC2 Run 1.1 . A lot of this was mostly in Twinkles, but that seems hard to work with. So, there is some re-implementation.
convenience functions

There is a convenience function in python/desc/sims/GCRCatSimInterface/InstanceCatalogWriter.py
snphosimcat : uses the catalog classes to give a phosim instance catalog which can be written out.

Demo :

There is an example of how to use the phosim catalog in Notebooks/demo_phosimcatalogs.ipynb`

Storage:

The way the current method works is that it writes out a filename of the phosim spectrum to the instance catalog, but the spectra are written out to spectra_files/Dynamic. We can change this to point to the path we like, and we can also make sure that we don't have to put the entire path on the phosim instance catalog row.

rbiswas4 commented 6 years ago

@danielsf You know most of the code as you helped me getting through this. One thing I was a little worried about was whether I had to do any database unlinking. Currently, it does not look that way. Could you please take a look at that?

@TomGlanzman Can you please see if the storage mechanism we discussed is implemented to your liking. This does put spectra in a directory different from the usual one. Currently the phosim instance catalog only has a file name (because I recall we preferred shorter names) but that can be easily changed.

@jbkalmbach : Do you have strongly lensed SN at high redshifts and do you need to see them for long times? Maybe you did it differently, but I had some trouble with that. I had to change the maxz and Time visible parameters as in python/desc/sims/GCRCatSimInterface/InstanceCatalogWriter.py

jchiang87 commented 6 years ago

@rbiswas4 I'm trying to understand how the SN sed files will be generated for Run1.1 as well as for later production runs that will use this code. It looks like the demo_phosimcatalogs.ipynb is meant to provide some guidance, but I think we either need a script in bin.src that does this in a self explanatory way (similar to create_agn_db.py) or we need real documentation, e.g., a fully annotated notebook or an .rst document. If I'm missing something, please let me know where I should be looking.

Also, operationally, this code doesn't seem to be hooked up to generateInstCat.py via InstanceCatalogWriter. When I run generateInstCat.py after merging from #17, the resulting instance catalogs look identical to what I would get from #17 as-is.

danielsf commented 6 years ago

I was able to get the InstanceCatalogWriter to run using this branch (I will explain how to run this test below). It runs, however:

The way we are constructing PhoSim InstanceCatalogs, we are separating each class of object (stars, galaxies, AGN) into separate catalog files and then including them in the main PhoSim catalog using PhoSim's includeobj command. The supernova catalogs do not get included this way. The catalog names need to be added to the object_catalogs kwarg of make_instcat_header in InstanceCatalogWriter.write_catalog().
The supernova catalog generator produces five distinct catalogs: -MainSurveyHostedSNPositions_cat_230.txt -MainSurvey_hostlessSN_cat_230.txt -MainSurvey_hostlessSN_highz_cat_230.txt -uDDFHostedSNPositions_cat_230.txt -uDDF_hostlessSN_cat_230.txt What are the differences between these? Which should actually be included in the InstanceCatalog?

To run this test, you will need the script workspace/instcat_writer/test_instance_catalog_writer.py which is on master. It will run through the whole InstanceCatalog generation process and write the results to workspace/instcat_writer/sprinkled/ (which it will create, if needed). The paths to the OpSim database and the agn_parameter database are hard-coded to copies that live in my SCRATCH space. I have given these 755 permissions, so if you just replace os.environ['SCRATCH'] with '/global/cscratch1/sd/danielsf' it should work. You also need to increase fov in write_catalog() to at least 0.5 to get non-empty supernova catalogs. You will also need to create the SED directory before running this script.

rbiswas4 commented 6 years ago

will try adding names of the SN catalogs.

The supernova catalog generator produces five distinct catalogs: -MainSurveyHostedSNPositions_cat_230.txt -MainSurvey_hostlessSN_cat_230.txt -MainSurvey_hostlessSN_highz_cat_230.txt -uDDFHostedSNPositions_cat_230.txt -uDDF_hostlessSN_cat_230.txt What are the differences between these? Which should actually be included in the InstanceCatalog?

According to what I have said (SNIa at the natural rate + enhanced in uDDF region but still not exceeding the natural rate of SN (ie if you consider core collapse) : all of them. However, I realize that a reasonable chunk comes from MainSurvey_hostlessSN_highz_cat_230.txt which is likely not very useful. So, if there is a crunch this could be left off without really affecting people. Hence I separated the highz and lowz. Aside from the optional nature of -MainSurvey_hostlessSN_highz_cat_230.txt (z > 1.0), everything should be there. The reason for having hostless SN is to try to have SN that do not have a galaxy background, on the same set of exposures (actually hopefully ccds) to compare during validation. The point is if things are bad at hosts that has a different source than being bad for hostles ones. The reason for having them in different catalogs is to have different objectId, and identify the origin easily.

I will try the test after fixing the paths.

rbiswas4 commented 6 years ago

@danielsf I made the changes but get permission denied to try and run your script.

rbiswas@cori04:~/src/DC2/sims_GCRCatSimInterface/workspace/instcat_writer> python test_instance_catalog_writer.py OPSIM /global/projecta/projectdirs/lsst/groups/SSim/DC2/minion_1016_desc_dithered_v4.db /global/cscratch1/sd/danielsf/proto_dc2_agn/test_agn.db Traceback (most recent call last): File "test_instance_catalog_writer.py", line 16, in assert os.path.exists(agn_db_name) AssertionError

jchiang87 commented 6 years ago

With the changes I've proposed I am able to get a set of phosim instance catalog files that look right. The following command line works at NERSC:

time generateInstCat.py --db /global/projecta/projectdirs/lsst/groups/SSim/DC2/minion_1016_desc_dithered_v4.db \
--agn_db_name /global/cscratch1/sd/jchiang8/DC2_prs/agn_db_mbh_7.0_m_i_30.0.sqlite \
--descqa_catalog protoDC2 \
--out_dir sprinkled_sn_sl_0.5 \
--ids 230 \
--fov 0.5 \
--protoDC2_ra 55.064 \
--protoDC2_dec -29.783 \
--enable_sprinkler

The code will need some fixing to produce an imsim catalog and will need some other fixes, but once my suggested changes are addressed, I think it will be ready for Run1.1p.

danielsf commented 6 years ago

Stupid thing: do we need to add twinkles to the ups/sims_GCRCatSimInterface.table file?

jchiang87 commented 6 years ago

With @rbiswas4 's most recent change, I was able to run the code to completion using the command line I posted last night. There are a bunch of formatting fixes I'd like to have done and refactoring to make everything work more coherently (and to fix the imsim catalog generation), but unless those issues prevent us from convincing ourselves that the current code will work for Run1.1p, I propose we merge. @danielsf Can you give the code a look? We may need to make another set of changes in case the SED file handling won't work for @TomGlanzman 's phosim pipeline, but I'd prefer to do that as an isolated change on a separate branch/pr.

jbkalmbach commented 6 years ago

Should we have a single variable to set somewhere that will tell the SN and sprinkled SN SEDs to go to the same place?

jbkalmbach commented 6 years ago

Or should we just wait to hear back from Tom and handle this all then?

danielsf commented 6 years ago

@jbkalmbach Probably, but I say merge and wait for feedback from Tom.

This branch has already been rebased too many times ;)

jchiang87 commented 6 years ago

Should we have a single variable to set somewhere that will tell the SN and sprinkled SN SEDs to go to the same place?

@jbkalmbach I'm not quite understanding this. I thought only the code that Rahul added in this branch is writing SEDs (to ./spectra_files/Dynamic). Are there other SED files being written somewhere?

jbkalmbach commented 6 years ago

Yes, if sprinkled SN exist the SEDs get written to the folder set here

jchiang87 commented 6 years ago

Yes, if sprinkled SEDs exist they get written to the folder set here

And the writing of those files is handled in the Twinkles classes? I assume that must be relative to the cwd, right? So the result will be a directory tree that looks like

spectra_files/
|-- Dynamic
|   |--specFileSN_1086_59580.1396_r.dat
|   |-- specFileSN_12039_59580.1396_r.dat
|   |-- specFileSN_12296_59580.1396_r.dat
...
|
`-- <Sprinkled SN files>

Is that right?

jbkalmbach commented 6 years ago

Yeah, that's what it should look like.

jchiang87 commented 6 years ago

ok, good. then I say we merge and make a new branch in case we need to adjust the SED writing.

TomGlanzman commented 6 years ago

I've a question about the SED files location. Referring to the argument descriptions in Jim's interface script, sims_GCRCatSimInterface/bin.src/generateInstCat.py, can someone describe in detail the structure of the "out_dir" created? I am interested in the names and locations of the main instanceCatalog file, included files, the SED directory naming and content. If a test has been run at NERSC for which I could examine the output, that would be great.

jchiang87 commented 6 years ago

Here is the resulting directory structure from that command:

[DC2_prs] tree sprinkled_sn_sl_0.5
sprinkled_sn_sl_0.5
|-- MainSurveyHostedSNPositions_cat_230.txt.gz
|-- MainSurvey_hostlessSN_cat_230.txt.gz
|-- MainSurvey_hostlessSN_highz_cat_230.txt.gz
|-- bright_stars_230.txt
|-- gal_cat_230.txt.gz
|-- knots_cat_230.txt
|-- phosim_cat_230.txt
|-- star_cat_230.txt.gz
|-- uDDFHostedSNPositions_cat_230.txt.gz
`-- uDDF_hostlessSN_cat_230.txt.gz

0 directories, 10 files

The instance catalogs are all in /global/cscratch1/sd/jchiang8/DC2_prs/sprinkled_sn_sl_0.5 and the SEDs are in /global/cscratch1/sd/jchiang8/DC2_prs/spectra_files

TomGlanzman commented 6 years ago

Thank you @jchiang87 and @rbiswas4 . In addition, Rahul provided me with a full example output and I observed an issue with the SED specification within the catalog files. We are slacking to hammer this out.

jchiang87 commented 6 years ago

ok, I will merge in ~30 minutes, unless I hear otherwise.

rbiswas4 commented 6 years ago

Scott pointed out a hole in documentation that I will fix. And, there is the question of whether you have a preference for storing the files that should not be there.

jchiang87 commented 6 years ago

Thanks @rbiswas4. ETA for merging is still 11:30 am PT.

rbiswas4 commented 6 years ago

@jchiang87 ready to merge

jchiang87 commented 6 years ago

ok, I'm doing a final git pull, will run the tests and the command line and if all looks ok, will merge....should be ~10mins.

TomGlanzman commented 6 years ago

Rahul and I have settled on a solution that will work for PhoSim (and the DC2 workflow). While he is working on the code and running a test, I will describe.

The "out_dir" specification will define a directory which will include (as it does now) all of the catalog files. In addition, it will contain a new subdirectory, "Dynamic" which will contain all of the SED files.

References to these dynamic SED files from the catalog files will look something like this:

object 3401774 53.4327981 -27.2598065 23.5489169 Dynamic/specFileSN_3322_59580.1396_r.dat 0.42006242 0 0 0 0 0 point CCM CCM 0.0275142564 3.1

In order to convince phoSim to find these files, I will create a symlink mirror of the production SED directory in $SCRATCH, e.g., $SCRATCH/DC2/\<visitID>/SEDs. Then, will add a single sym link named "Dynamic" from there to the out_dir/Dynamic directory. Finally, phoSim will be run specifying the $SCRATCH area for all of its SEDs. This scheme will allow an arbitrary number of simultaneous and distinct phosim instances to run in parallel.

jchiang87 commented 6 years ago

@TomGlanzman Can you make a new github issue for that discussion please? Or wait for Rahul's PR and put that text there? Thanks.

jchiang87 commented 6 years ago

So after doing the git pull, I tried to rerun that command line and the code is failing with sqlalchemy errors. I don't see how the last two commits could possibly cause an issue, but I think we need to get this working before merging.

jchiang87 commented 6 years ago

ceback (most recent call last):
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/cscratch1/sd/jchiang8/DC2_prs/sims_GCRCatSimInterface/bin/generateInstCat.py", line 51, in <module>
    instcat_writer.write_catalog(obsHistID, out_dir=args.out_dir, fov=args.fov)
  File "/global/cscratch1/sd/jchiang8/DC2_prs/sims_GCRCatSimInterface/python/desc/sims/GCRCatSimInterface/InstanceCatalogWriter.py", line 287, in write_catalog
    write_header=False)
  File "/global/cscratch1/sd/jchiang8/DC2_prs/sims_GCRCatSimInterface/python/desc/sims/GCRCatSimInterface/TwinklesClasses.py", line 173, in write_catalog
    write_mode=write_mode)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sims_catalogs/2.4.2.sims/python/lsst/sims/catalogs/definitions/CompoundInstanceCatalog.py", line 355, in _write_compound
    for chunk in master_results:
  File "/global/cscratch1/sd/jchiang8/DC2_prs/sims_GCRCatSimInterface/python/desc/sims/GCRCatSimInterface/DatabaseEmulator.py", line 140, in __next__
    return self._descqa_obj._postprocess_results(chunk)
  File "/global/cscratch1/sd/jchiang8/DC2_prs/sims_GCRCatSimInterface/python/desc/sims/GCRCatSimInterface/ProtoDC2DatabaseEmulator.py", line 295, in _postprocess_results
    chunk_size=1000000)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sims_catalogs/2.4.2.sims/python/lsst/sims/catalogs/db/dbConnection.py", line 470, in get_arbitrary_chunk_iterator
    return self.get_chunk_iterator(query, chunk_size = chunk_size, dtype = dtype)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sims_catalogs/2.4.2.sims/python/lsst/sims/catalogs/db/dbConnection.py", line 484, in get_chunk_iterator
    return ChunkIterator(self, query, chunk_size, arbitrarySQL = True)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sims_catalogs/2.4.2.sims/python/lsst/sims/catalogs/db/dbConnection.py", line 78, in __init__
    self.exec_query = dbobj.connection.session.execute(query)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/scoping.py", line 150, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 1023, in execute
    bind, close_with_result=True).execute(clause, params or {})
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 914, in execute
    return meth(self, multiparams, params)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1146, in _execute_context
    context)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 181, in reraise
    raise value.with_traceback(tb)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1139, in _execute_context
    context)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 450, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file [SQL: 'SELECT galaxy_id, magNorm, varParamStr FROM agn_params WHERE galaxy_id BETWEEN 10936 AND 11311777 ORDER BY galaxy_id']
Exception during reset or similar
Traceback (most recent call last):
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 668, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 420, in do_rollback
    dbapi_connection.rollback()
sqlite3.ProgrammingError: Cannot operate on a closed database.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 636, in _finalize_fairy
    fairy._reset(pool)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 774, in _reset
    self._reset_agent.rollback()
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1563, in rollback
    self._do_rollback()
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1601, in _do_rollback
    self.connection._rollback_impl()
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 670, in _rollback_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 181, in reraise
    raise value.with_traceback(tb)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 668, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 420, in do_rollback
    dbapi_connection.rollback()
sqlalchemy.exc.ProgrammingError: <unprintable ProgrammingError object>
Exception during reset or similar
Traceback (most recent call last):
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 668, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 420, in do_rollback
    dbapi_connection.rollback()
sqlite3.ProgrammingError: Cannot operate on a closed database.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 636, in _finalize_fairy
    fairy._reset(pool)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/pool.py", line 774, in _reset
    self._reset_agent.rollback()
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1563, in rollback
    self._do_rollback()
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1601, in _do_rollback
    self.connection._rollback_impl()
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 670, in _rollback_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1341, in _handle_dbapi_exception
    exc_info
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 188, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=exc_value)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/util/compat.py", line 181, in reraise
    raise value.with_traceback(tb)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 668, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
  File "/global/common/software/lsst/cori-haswell-gcc/stack/w.2017.46_py3_gcc6/stack/miniconda3-4.3.21-10a4fa6/Linux64/sqlalchemy/1.0.8.lsst3+4/lib/python/SQLAlchemy-1.0.8-py3.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 420, in do_rollback
    dbapi_connection.rollback()
sqlalchemy.exc.ProgrammingError: <unprintable ProgrammingError object>

real    3m27.158s
user    1m13.383s
sys 0m18.401s

danielsf commented 6 years ago

@jchiang87 ....grumble... I suspect this on the lsst_sims side.

I will try to fix it.

jchiang87 commented 6 years ago

I think I see the issue actually....some files have appeared in my directory with Rahul's uid and gid for the agn params db.

danielsf commented 6 years ago

Oh good... There is a lunch talk here I would like to attend.

I will check back in at 1:30 PST to see if a rush fix is needed

jchiang87 commented 6 years ago

The issue was temporary files created by sqlite that were hanging around under Rahul's userid. (see https://sqlite.org/tempfiles.html for possible mitigations in the future). The test ran fine after cleaning up those files. So proceeding with the merge.