Tracking issue for 0.7.0

[lgray]

• [x] remove awkward0 and related coffea subpackages (analysis_objects, arrays, nanoaod)
• [x] minimize use of uproot3
• [x] resurrect all pertinent tests
• [x] accuracy tests in addition to functionality tests for ak1 jetmet tools (possibly backport as well)
• [x] migrate JME tools to newer/common lookup tools backends
• [x] update binder examples and any ak0 specific documentation

@aperloff @kondratyevd @jrueb - if you want to help with the accuracy tests for JME I'd really appreciate it.

Most of these are handled in #386, and this is likely incomplete. @nsmith- @areinsvo @mcremone please feel free to add items here, and we can discuss further in comments.

Possible additions:

• refactorization of parts of nanoevents (i.e. methods package can be moved up a level, do we like the way array access is done right now)
• more factored out dealing with arrow and parquet
• better dealing with pandas in spark

[lgray]

@nsmith- can you check there's no documentation referencing ak0 specifically? Then we can check off the last item.

[ivotron]

hi @nsmith- and @lgray , quick question/follow-up regarding arrow/parquet. Will it be possible to read parquet (nanoevent) files through arrow's dataset API? thanks!

[lgray]

If you can make an example of the data that's generated we can check? If the dataset api spits out arrow tables you're likely very close to done.

[ivotron]

we actually don't have data in parquet already. Is it possible to convert from root to parquet and then read those?

[lgray]

Yep, you should be able to sanely transform root into parquet with uproot + awkward now. The issues with correctly preserving nullability are fixed (so far as we know).

NanoAOD shouldn't be a huge deal.

[nsmith-]

Looked around for awkward 0 stuff, and only found an old link and updated it in https://github.com/CoffeaTeam/coffea/pull/403/commits/5afc91815b256366f8e988ef39757347f54b5359

[lgray]

awkward0 doesn't convert things in a sane way for our purposes (the nullability and some other stuff is all wrong).

Probably better to cook an awkward1 converter and stick it in util.

[matthewfeickert]

:wave: Hi. It seems that coffea is currently the only thing that is keeping uproot3, uproot3_methods, and awkward0 in LCG dev4[^1] (c.f. https://sft.its.cern.ch/jira/browse/SPI-2229?focusedCommentId=115655&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-115655). Is uproot3 something that is still deeply depended on by coffea v0.7.19+ (I see that there was work to minimize this in PR #386) or is this something that with some contributions from IRIS-HEP people could be removed and moved towards uproot v4?

[^1]: Let it be noted that I don't generally suggest that people go for LCG views if they don't have to, but I'm trying to update a bunch of Scikit-HEP/IRIS-HEP packages in LCG dev4 in general.

[lgray]

@matthewfeickert uproot3 is not that deeply depended on but provided some expected behaviors for certain folks. It will go away in coffea 0.8.0 (~start of 2023).

[nsmith-]

It is only for dumping coffea hist into ROOT files: https://github.com/CoffeaTeam/coffea/blob/ec7601b0cc09b9f2bb9217f7cec8d6d6d15a2256/coffea/hist/export.py#L38-L43 Unfortunately, the current API assumes the user is creating an output file in uproot3 so it cannot just be silently upgraded to uproot4. Perhaps we can put the needed import and class definitions inside the function to at least remove the dependency on install.

[lgray]

All of coffea.hist is being removed, so we just stuck that in an uproot 4/5 style implementation in util.