Add four more sources of constants data - Wikidata, NIST Webbook, Common Chemistry, and the Joback group contribution method

[CalebBell]

Creating a PR to observe if tests pass on other platforms; not ready yet.

[CalebBell]

Hi Yoel, This PR adds the following data sources:

• Constant properties from the NIST WebBook (all species)
• Melting and boiling points plus densities from Common Chemistry (3/5 of all species in their free database, all that I found)
• Constant properties from Wikidata
• Estimated properties from the Joback group contribution method
• A source of critical properties, Piña-Martinez, Andrés, Romain Privat, and Jean-Noël Jaubert. “Use of 300,000 Pseudo-Experimental Data over 1800 Pure Fluids to Assess the Performance of Four Cubic Equations of State: SRK, PR, Tc-RK, and Tc-PR.” AIChE Journal n/a, no. n/a (n.d.): e17518. https://doi.org/10.1002/aic.17518.

It makes the following technical changes:

• The option to store the dataframes in memory as sparse arrays
• The option to store dataframes using integers as indexes, instead of strings. This is about 5x as memory efficient as using strings. This is turned on only for the new data sources.
• Formulas with unicode subscripts ₀₁₂₃₄₅₆₇₈₉ are now handled by the formula parsing code.

Also included are the following scripts:

• /dev/data/update_wikidata.py downloads all of the available data from Wikidata. It takes only a couple of minutes, and I spend quite a lot of time working on it. One feature missing in Chemicals is the ability for the user to add their own data. This script makes that a little possible - if someone asks for a chemical's data to be included and has values, they can add them to Wikidata, and we can re-run the script and release a new version of Chemicals with the new data.
• /dev/data/update_common_chemistry.py downloads and processes the data from common chemistry, writing the output file. This script takes incredibly long to run, but caches downloaded data so future runs are much quicker. The file /dev/data/common_chemistry_CASs.tsv contains as many CAS numbers in their free database as I could find (315817); they claim to have almost 500,000 free chemicals, but I am out of ideas for finding more of them. It is not very likely their data will be expanded in the future, so re-running this one will probably not be necessary in the future.
• /dev/data/generate_joback_predictions.py generates the Joback predictions file.

A script for NIST WebBook is not yet available, and I hope to add further data for non-constant properties from them as well.

Could you give this PR a quick look and let me know what you think? I am trying to expand the number of chemicals with available data.

Sincerely, Caleb

[yoelcortes]

Hi Caleb,

I see you have been busy! These updates all look very exciting. I'll have a detailed look within the next day.

Thanks!

[yoelcortes]

Hi Caleb! The updates look great overall!

I made some minor changes to prevent attempting CAS_to_int if the user passes an integer (which is an update hope to make in thermo during chemical creation). Let me know if there are any issues with my implementation.

I added a "low memory" run to the github workflow, which shows some issues with the test. A few of the issues are trivial (like dataframe shapes not matching), but others may need a little more time to solve. But all the tests are passing without "low memory".

I'll be flying back to the US and will be a little busy until January 7, so I won't get a chance to address the test issues until then.

Wish you happy holidays and a happy new year! Thanks,

[CalebBell]

Hello Yoel!

Thank you very much for the multitude of improvements. If I understand what you are after, you would like to be able to do "Tc(CASRN=64175)" and get the critical temperature? I think that is pretty doable with a few more changes if so. Updating the documentation and adding a few more tests is probably the hardest part to making that happen. I'm glad you noticed the CHEDL_LOW_MEMORY environmental variable I was playing around with. I was curious how much memory could be saved if more strings were removed, so I added a flag. It wasn't my intention to be providing this as a feature to users. The flag saves about 1/3 of the memory used, or 6 MB. Longer term, I hope to be able to reduce memory use for most users in the following two ways:

• Adding a "best available" data source, which contains the default (quasi-recommended) value for every property for every chemical for which the data is available. If like most users they aren't specifying a method, this will work fine.
• Breaking up the lazy-loading to be even more granular, so each data frame is loaded only when it needs to be searched; instead of all dataframes for a particular property being loaded first.

Your point about the tests not passing with that variable set was expected for me. The flag essentially removes some of the functionality of the library, so some of the tests should be expected to fail. I was able to make them pass through a few fixes and replacing the previously deleted column with an empty sparse column. I like the pytest option to run the tests with a changed environmental variable; nice! I definitely could have communicated better what I was trying to do with the flag though. With my hopefully-better explanation, do you think it is still necessary to add the CI for the low memory mode? I wouldn't want to update documentation and point users towards this flag at this point.

I wish you a happy holidays and new year as well! Caleb

[yoelcortes]

Hi Caleb,

OK! I like the flag and may use it myself in the near future. The points you mentioned on reducing memory use would be wonderful too.

It's great that you were able to get the tests passing with low memory! I agree with not updating documentation and pointing users towards the new flag yet. If we intend to keep the CHEDL_LOW_MEMORY variable, I think keeping the flag in the CI may be better in the long run. But if there is a good chance we remove it in favor of something else, we can remove it from the CI (your call).

Thinking for BioSTEAM, I might include a function to delete all the dataframes in chemicals after loading the thermodynamic property packages. The function would also set "_loaded" attributes to false. This would save almost 100% of the memory. But using the flag may be useful for more dynamic efforts to keep the dataframes.

Thanks!