Closed r0mainK closed 4 years ago
@vmarkovtsev this is rebased and modified according to the first part. In order to avoid some redundancy between siva.go
and uast.go
I replaced the ForEach
method with a ForEachRepository
function that takes the dataset
as input. This allowed me to refactor Uast/SivaRepositoryFromCSV
functions in a common RepositoryFromCSV
interface method.
Just added a couple commits: temporary files were not cleaned in case of command cancellation - furthermore the change I brought to stop updating the Bar
in case of an issue with pga get
also caused files to not be cleaned, in the case multiple files were being downloaded.
For more info on context see this ml issue and this infra issue.
This PR only deals with the first dataset, ie the UASTs extracted from the HEAD of PGAv2, and stored as parquet. It does not contain yet any documentation, apart from in side the code - and I did not put much effort into the comments, as I wasn't sure the logic would be kept. Anyway here's a run down of the changes:
commit 1: typos
commit 2 and 3:
Add
uast_dataset.go
andsiva_dataset.go
. Basically, since both datasets are comprised of individual files I wanted to keep the same commands for listing and downloading, so I had to abstract all the logic fromIndexToCSV
, RepositoryFromCSV andToCSV
, as well as the schemas. In order to do that I:Repository
in an interface withToCSV
as it's methodIndexFromCSV
as it's methodRepositoryFromCSV
to be dataset specific, and handled by dataset specific index - as we don't need to deal with legacy indexes for the new dataset.Repository
'sURL
,Languages
andSiva
/Parquet
Filenames
, used in other places.commit 4
Following the previous changes, I basically removed all code I move to
siva_dataset
, added parsers/formatter forfloats
with 2 point precision as well as acsvColumn
field, and created a mapping for datasets. Also modified the types asRepository
is now an interface, and (I think) can't be pointed to the same.commit 5
Now using getters in
filters.go
, and updated typing.commit 6
Added a handler for the
datasetName
arg now used inget
/list
.commit 7
Updated commands to the new structure, more of the same: getters, typing, now using
Dataset
andDataset
handler.Note: I was thinking of adding a parquet command to dump listing of individual parquet files, as well as some additional filters to be used only by the
uast
dataset, related to the extraction rate. What do you think ?