search

tardis-sn / carsus

Atomic Database for Astronomy
https://tardis-sn.github.io/carsus
21 stars 44 forks source link

Extending the atomic database #112

Closed jselsing closed 4 years ago

jselsing commented 6 years ago

Hi all

I'm keen on trying to run TARDIS with heavier elements than the ones included in the default distribution (Up to Z = 30).

I have installed TARDIS with "pip install git+https://github.com/tardis-sn/tardis“ and have set up the tardis conda environments, as suggested as using the depencecy.yml from "https://raw.githubusercontent.com/tardis-sn/tardis/master/tardis_env27.yml”. However, scipy and pyne had dependency clashes and only after updating all the required packages with (conda update —all) can TARDIS successfully do the spectral synthesis.

Similarly, I have installed carsus with "pip install git+https://github.com/tardis-sn/carsus“ and all required packages. I have install ChiantiPy and downloaded the Chianti database and carsus-db.

Running the notebook, https://github.com/tardis-sn/carsus/blob/master/docs/notebooks/kurucz_chianti_h_he.ipynb, throws this error when running the Ionization Energies box:

Downloading ionization energies from the NIST Atomic Spectra Database
/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.py:89: ParserWarning: Falling back to the 'python' engine because the 'c' engine does not support skipfooter; you can avoid this warning by specifying engine='python'.
  usecols=range(5), names=column_names, skiprows=3, skipfooter=1)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-b697c495f097> in <module>()
      5 ioniz_energies_ingester.ingest(
      6             ionization_energies=True,
----> 7             ground_levels=True
      8             )
      9 session.commit()

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.pyc in ingest(self, ionization_energies, ground_levels)
    287         # Download data if needed
    288         if self.parser.base is None:
--> 289             self.download()
    290 
    291         if ionization_energies:

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.pyc in download(self)
    221     def download(self):
    222         data = self.downloader(spectra=self.spectra)
--> 223         self.parser(data)
    224 
    225     def ingest_ionization_energies(self, ioniz_energies=None):

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/base.pyc in __call__(self, input_data)
     42 
     43     def __call__(self, input_data):
---> 44         self.load(input_data)
     45 
     46 

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.pyc in load(self, input_data)
     89                          usecols=range(5), names=column_names, skiprows=3, skipfooter=1)
     90         for column in ['ground_shells', 'ground_level', 'ionization_energy_str']:
---> 91                 base[column] = base[column].map(lambda x: x.strip())
     92         self.base = base
     93         # column_names = ['atomic_number', 'ion_charge', 'ground_shells', 'ground_level', 'ionization_energy_str', 'ionization_energy_uncertainty_str']

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/pandas/core/series.pyc in map(self, arg, na_action)
   2352         else:
   2353             # arg is a function
-> 2354             new_values = map_f(values, arg)
   2355 
   2356         return self._constructor(new_values,

pandas/_libs/src/inference.pyx in pandas._libs.lib.map_infer()

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/nist/ionization.pyc in <lambda>(x)
     89                          usecols=range(5), names=column_names, skiprows=3, skipfooter=1)
     90         for column in ['ground_shells', 'ground_level', 'ionization_energy_str']:
---> 91                 base[column] = base[column].map(lambda x: x.strip())
     92         self.base = base
     93         # column_names = ['atomic_number', 'ion_charge', 'ground_shells', 'ground_level', 'ionization_energy_str', 'ionization_energy_uncertainty_str']

AttributeError: 'NoneType' object has no attribute 'strip'

I think this is due to the variable number of footer lines that are returned from the NIST query.

If I change the pandas.read_csv call in:

https://github.com/tardis-sn/carsus/blob/master/carsus/io/nist/ionization.py

from

base = pd.read_csv(StringIO(text_data), sep='|', header=None, usecols=range(5), names=column_names, skiprows=3, skipfooter=1)

to:

base = pd.read_csv(StringIO(text_data), sep='|', header=None, usecols=range(5), names=column_names, skiprows=3).dropna(how = "any")

correctly chops away the rows containing NaN values and the NIST part can be concluded. However, when ingesting the Kurucz database (GFALLIngester), the line-list are not correctly connected and it throws a SQL-related error my SQL-fu is not strong enough to clear up. 

Ingesting levels from ku_latest
Ingesting levels for He 0
Ingesting levels for He 1
Ingesting levels for Li 0
Ingesting levels for Li 1
Ingesting levels for Be 0
Ingesting levels for Be 1
Ingesting levels for Be 2
Ingesting lines from ku_latest
Ingesting lines for He 0
Ingesting lines for He 1
---------------------------------------------------------------------------
InterfaceError                            Traceback (most recent call last)
<ipython-input-4-3d924feb90ac> in <module>()
      1 gfall_ingester = GFALLIngester(session, gfall_fname, ions='H-Be')
----> 2 gfall_ingester.ingest(levels=True, lines=True)
      3 session.commit()

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/kurucz/gfall.py in ingest(self, levels, lines)
    435 
    436         if lines:
--> 437             self.ingest_lines()
    438             self.session.flush()
    439 

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/kurucz/gfall.py in ingest_lines(self, lines)
    394             print("Ingesting lines for {} {}".format(convert_atomic_number2symbol(atomic_number), ion_charge))
    395             # print(ion)
--> 396             lvl_index2id = self.get_lvl_index2id(ion)
    397 
    398             for index, row in ion_lines.iterrows():

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/carsus/io/kurucz/gfall.py in get_lvl_index2id(self, ion)
    328 
    329         lvl_index2id = list()
--> 330         for id, index in q_ion_lvls:
    331             lvl_index2id.append((index, id))
    332 

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/query.pyc in __iter__(self)
   2875         context.statement.use_labels = True
   2876         if self._autoflush and not self._populate_existing:
-> 2877             self.session._autoflush()
   2878         return self._execute_and_instances(context)
   2879 

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in _autoflush(self)
   1442                     "consider using a session.no_autoflush block if this "
   1443                     "flush is occurring prematurely")
-> 1444                 util.raise_from_cause(e)
   1445 
   1446     def refresh(

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/util/compat.pyc in raise_from_cause(exception, exc_info)
    201     exc_type, exc_value, exc_tb = exc_info
    202     cause = exc_value if exc_value is not exception else None
--> 203     reraise(type(exception), exception, tb=exc_tb, cause=cause)
    204 
    205 if py3k:

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in _autoflush(self)
   1432         if self.autoflush and not self._flushing:
   1433             try:
-> 1434                 self.flush()
   1435             except sa_exc.StatementError as e:
   1436                 # note we are reraising StatementError as opposed to

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in flush(self, objects)
   2241         try:
   2242             self._flushing = True
-> 2243             self._flush(objects)
   2244         finally:
   2245             self._flushing = False

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in _flush(self, objects)
   2367         except:
   2368             with util.safe_reraise():
-> 2369                 transaction.rollback(_capture_exception=True)
   2370 
   2371     def bulk_save_objects(

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.pyc in __exit__(self, type_, value, traceback)
     64             self._exc_info = None   # remove potential circular references
     65             if not self.warn_only:
---> 66                 compat.reraise(exc_type, exc_value, exc_tb)
     67         else:
     68             if not compat.py3k and self._exc_info and self._exc_info[1]:

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/session.pyc in _flush(self, objects)
   2331             self._warn_on_events = True
   2332             try:
-> 2333                 flush_context.execute()
   2334             finally:
   2335                 self._warn_on_events = False

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.pyc in execute(self)
    389                     self.dependencies,
    390                     postsort_actions):
--> 391                 rec.execute(self)
    392 
    393     def finalize_flush_changes(self):

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/unitofwork.pyc in execute(self, uow)
    554                              uow.states_for_mapper_hierarchy(
    555                                  self.mapper, False, False),
--> 556                              uow
    557                              )
    558 

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/persistence.pyc in save_obj(base_mapper, states, uowtransaction, single)
    179         _emit_insert_statements(base_mapper, uowtransaction,
    180                                 cached_connections,
--> 181                                 mapper, table, insert)
    182 
    183     _finalize_insert_update_commands(

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/orm/persistence.pyc in _emit_insert_statements(base_mapper, uowtransaction, cached_connections, mapper, table, insert, bookkeeping)
    864                 else:
    865                     result = cached_connections[connection].\
--> 866                         execute(statement, params)
    867 
    868                 primary_key = result.context.inserted_primary_key

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in execute(self, object, *multiparams, **params)
    946             raise exc.ObjectNotExecutableError(object)
    947         else:
--> 948             return meth(self, multiparams, params)
    949 
    950     def _execute_function(self, func, multiparams, params):

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/sql/elements.pyc in _execute_on_connection(self, connection, multiparams, params)
    267     def _execute_on_connection(self, connection, multiparams, params):
    268         if self.supports_execution:
--> 269             return connection._execute_clauseelement(self, multiparams, params)
    270         else:
    271             raise exc.ObjectNotExecutableError(self)

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in _execute_clauseelement(self, elem, multiparams, params)
   1058             compiled_sql,
   1059             distilled_params,
-> 1060             compiled_sql, distilled_params
   1061         )
   1062         if self._has_events or self.engine._has_events:

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in _execute_context(self, dialect, constructor, statement, parameters, *args)
   1198                 parameters,
   1199                 cursor,
-> 1200                 context)
   1201 
   1202         if self._has_events or self.engine._has_events:

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in _handle_dbapi_exception(self, e, statement, parameters, cursor, context)
   1411                 util.raise_from_cause(
   1412                     sqlalchemy_exception,
-> 1413                     exc_info
   1414                 )
   1415             else:

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/util/compat.pyc in raise_from_cause(exception, exc_info)
    201     exc_type, exc_value, exc_tb = exc_info
    202     cause = exc_value if exc_value is not exception else None
--> 203     reraise(type(exception), exception, tb=exc_tb, cause=cause)
    204 
    205 if py3k:

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/base.pyc in _execute_context(self, dialect, constructor, statement, parameters, *args)
   1191                         statement,
   1192                         parameters,
-> 1193                         context)
   1194         except BaseException as e:
   1195             self._handle_dbapi_exception(

/usr/local/anaconda3/envs/tardis/lib/python2.7/site-packages/sqlalchemy/engine/default.pyc in do_execute(self, cursor, statement, parameters, context)
    505 
    506     def do_execute(self, cursor, statement, parameters, context=None):
--> 507         cursor.execute(statement, parameters)
    508 
    509     def do_execute_no_params(self, cursor, statement, context=None):

InterfaceError: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (sqlite3.InterfaceError) Error binding parameter 1 - probably unsupported type. [SQL: u'INSERT INTO transition (type, lower_level_id, upper_level_id, data_source_id) VALUES (?, ?, ?, ?)'] [parameters: ('line',           id
index       
0      12253
0      53998,           id
index       
764    13017
764    54762, 3)] (Background on this error at: http://sqlalche.me/e/rvf5)

It looks like it correctly gets the lines for the neutral species, but complains when trying the singly ionised. Maybe it is not correctly separating the ions by charge?

epassaro commented 4 years ago

This is probably fixed now.