Prevent default data unzipping when sourceType is 'other'

eblondel commented 1 year ago

Through the entity$copyDataToJobDir method, zipped sources are unzipped, essential for cases of spatial files for which data has to be read to inherit data properties and propagate them in metadata. For sources tagged with sourceType other, it is not needed by default.

As a first approach, a zip archive should be kept as it is when it is tagged as "other" type.

@juldebar @kikislater

eblondel commented 1 year ago

@kikislater i've pushed some code to change the behaviour for data with sourceType:other. The approach is valid for all ways to access the data, including by means of data accessors (with property access)

kikislater commented 1 year ago

Thank you but doesn't work =>

Data column, first try:

"source:session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip@/home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip_
sourceType:other"

Answer:

[geoflow][zenodo][INFO] Copying data source 1 'session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip' (/home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip) to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230802165121/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' 
[geoflow][zenodo][INFO] Copying data to Job data directory from local file(s) 
[geoflow][zenodo][INFO] Copying data local file(s): copying unzipped files to job data directory 
[geoflow][zenodo][INFO] sourceZip = FALSE: source files will be uploaded 
[geoflow][zenodo][WARN] Metadata dynamic handling based on 'data' not implemented for source type 'other'

Adding sourceZip:true Answer:

[geoflow][zenodo][INFO] Copying data source 1 'session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip' (/home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip) to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230802165508/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' 
[geoflow][zenodo][INFO] Copying data to Job data directory from local file(s) 
[geoflow][zenodo][INFO] Copying data local file(s): copying unzipped files to job data directory 
[geoflow][zenodo][INFO] sourceZip = TRUE: Zip sources into single data file 
[1] "DCIM"     "METADATA"
[geoflow][zenodo][INFO] sourceZipOnly = FALSE: both zip and zipped files will be uploaded 
[geoflow][zenodo][WARN] Metadata dynamic handling based on 'data' not implemented for source type 'other'

Adding sourceZipOnly:true Answer:

[geoflow][zenodo][INFO] Copying data source 1 'session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip' (/home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip) to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230802165708/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' 
[geoflow][zenodo][INFO] Copying data to Job data directory from local file(s) 
[geoflow][zenodo][INFO] Copying data local file(s): copying unzipped files to job data directory 
[geoflow][zenodo][INFO] sourceZip = TRUE: Zip sources into single data file 
[1] "DCIM"     "METADATA"
[geoflow][zenodo][INFO] sourceZipOnly = TRUE: deleting zipped, they will not be uploaded 
[geoflow][zenodo][WARN] Metadata dynamic handling based on 'data' not implemented for source type 'other'

eblondel commented 1 year ago

That's weird, are you sure the new geoflow has been properly installed?

kikislater commented 1 year ago

Yes, I removed cached files in /tmp/ and checked the new code downloaded, especially line 498 of geoflow_entity.R before build Please note that there are only folders in Zip. Files are available in folder.

eblondel commented 1 year ago

i've added some log to understand what is the sourceType in your case. We have some business logic that transform this sourceType, but this is done later in the code when we try to enrich the entity with spatial properties coming from spatial data. In this case we just look into zip files to detect spatial data files. There might be the need to consolidate codes in this business logic, but first I want to understand why your sourceType is altered, and the control I've added doesn't work.

kikislater commented 1 year ago

From log I got this :

====================================================================================================
Workflow Execution
====================================================================================================
[geoflow][zenodo][INFO] Executing workflow job... 
----------------------------------------------------------------------------------------------------
[geoflow][zenodo][INFO] Executing software actions 'onstart' ... 
----------------------------------------------------------------------------------------------------
[geoflow][zenodo][INFO] Executing entity actions ... 
[geoflow][zenodo][INFO] Workflow mode: entity 
[geoflow][zenodo][INFO] Workflow with 1 actions 
[geoflow][zenodo][INFO] Action 1: zen4R-deposit-record 
[geoflow][zenodo][INFO] Create entity job dir at '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2' 
[geoflow][zenodo][INFO] Creating 'data' directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data 
[geoflow][zenodo][INFO] Creating 'metadata' directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/metadata 
[geoflow][zenodo][INFO] Entity working directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2 
[geoflow][zenodo][INFO] Copying data to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' 
[geoflow][zenodo][INFO] Copying data source 1 'session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip' (/home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip) to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' 
[geoflow][zenodo][INFO] Copying data to Job data directory from local file(s) 
[geoflow][zenodo][WARN] Source type: other 
[geoflow][zenodo][INFO] sourceZip = TRUE: Zip sources into single data file 
character(0)
[geoflow][zenodo][INFO] sourceZipOnly = TRUE: deleting zipped, they will not be uploaded 
[geoflow][zenodo][WARN] Metadata dynamic handling based on 'data' not implemented for source type 'other'

And from output :

executeWorkflow("20230727-133457_generated-config.json")
Rows: 121 Columns: 14
── Column specification ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (14): Identifier, Email, OrganizationName, PositionName, LastName, FirstName, PostalAddress, PostalCode, City, Country, Voice, Facs...

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 1 Columns: 15
── Column specification ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (15): Identifier, Title, Description, Subject, Creator, Date, Type, Language, SpatialCoverage, TemporalCoverage, Relation, Rights, ...

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Error in zip::zip_list(file.path(getwd(), paste0(basefilename, ".zip"))) : 
  Cannot open zip file `/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip`
De plus : Message d'avis :
Dans normalizePath(zipfile) :
  path[1]="/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip": Aucun fichier ou dossier de ce type
Erreur dans withVisible(...elt(i)) : 
  Workflow failed during execution, check logs at: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/job-logs.txt

So now it doesn't copy the content of zip or zip itself. It creates an empty zip with a curious filename:

ls -lha /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2_files_for_source_1.zip 
-rw-r--r-- 1 sylvain sylvain 22  4 août  21:09 /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2_files_for_source_1.zip

Edit: of course this zip is available on disk

[geoflow][zenodo][INFO] Copying data source 1 'session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip' (/home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip) to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data'

e.g.:

$ ls -lha /home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip
-rw-r--r-- 1 sylvain sylvain 943K  4 août  19:34 /home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip

kikislater commented 1 year ago

So filename is set from this line: https://github.com/r-geoflow/geoflow/blob/b169ca48d0c201fcd06fa8bd3ea2a0a35e6b1a40/R/geoflow_entity.R#L549

eblondel commented 1 year ago

You say that you dont want to have your archive unzip, please try to set sourceZip and sourceZipOnly to FALSE, this tries to zip files, but in your case it doesnt apply ( surely i miss some controls)

Le ven. 4 août 2023 à 19:16, Sylvain POULAIN @.***> a écrit :

From log I got this :

==================================================================================================== Workflow Execution

[geoflow][zenodo][INFO] Executing workflow job...

[geoflow][zenodo][INFO] Executing software actions 'onstart' ...

[geoflow][zenodo][INFO] Executing entity actions ... [geoflow][zenodo][INFO] Workflow mode: entity [geoflow][zenodo][INFO] Workflow with 1 actions [geoflow][zenodo][INFO] Action 1: zen4R-deposit-record [geoflow][zenodo][INFO] Create entity job dir at '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2' [geoflow][zenodo][INFO] Creating 'data' directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data [geoflow][zenodo][INFO] Creating 'metadata' directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/metadata [geoflow][zenodo][INFO] Entity working directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2 [geoflow][zenodo][INFO] Copying data to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' [geoflow][zenodo][INFO] Copying data source 1 'session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip' (/home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip) to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' [geoflow][zenodo][INFO] Copying data to Job data directory from local file(s) [geoflow][zenodo][WARN] Source type: other [geoflow][zenodo][INFO] sourceZip = TRUE: Zip sources into single data file character(0) [geoflow][zenodo][INFO] sourceZipOnly = TRUE: deleting zipped, they will not be uploaded [geoflow][zenodo][WARN] Metadata dynamic handling based on 'data' not implemented for source type 'other'

And from output :

executeWorkflow("20230727-133457_generated-config.json") Rows: 121 Columns: 14 ── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Delimiter: "," chr (14): Identifier, Email, OrganizationName, PositionName, LastName, FirstName, PostalAddress, PostalCode, City, Country, Voice, Facs...

ℹ Use spec() to retrieve the full column specification for this data. ℹ Specify the column types or set show_col_types = FALSE to quiet this message. Rows: 1 Columns: 15 ── Column specification ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Delimiter: "," chr (15): Identifier, Title, Description, Subject, Creator, Date, Type, Language, SpatialCoverage, TemporalCoverage, Relation, Rights, ...

ℹ Use spec() to retrieve the full column specification for this data. ℹ Specify the column types or set show_col_types = FALSE to quiet this message. Error in zip::zip_list(file.path(getwd(), paste0(basefilename, ".zip"))) : Cannot open zip file /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip De plus : Message d'avis : Dans normalizePath(zipfile) : path[1]="/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip": Aucun fichier ou dossier de ce type Erreur dans withVisible(...elt(i)) : Workflow failed during execution, check logs at: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/job-logs.txt

So now it doesn't copy the content of zip or zip itself. It creates an empty zip:

ls -lha /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2_files_for_source_1.zip -rw-r--r-- 1 sylvain sylvain 22 4 août 21:09 /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230804210935/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2_files_for_source_1.zip

— Reply to this email directly, view it on GitHub https://github.com/r-geoflow/geoflow/issues/344#issuecomment-1665941758, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKDK3HJOVCYRKAQJX5ZVJ3XTUVARANCNFSM6AAAAAA3BCNVMQ . You are receiving this because you authored the thread.Message ID: @.***>

kikislater commented 1 year ago

ok but already tested with everything empty in jobs data folder:

executeWorkflow("20230727-133457_generated-config.json")
Rows: 121 Columns: 14
── Column specification ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (14): Identifier, Email, OrganizationName, PositionName, LastName, FirstName, PostalAddress, PostalCode, City, Country, Voice, Facs...

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Rows: 1 Columns: 15
── Column specification ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Delimiter: ","
chr (15): Identifier, Title, Description, Subject, Creator, Date, Type, Language, SpatialCoverage, TemporalCoverage, Relation, Rights, ...

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
Error in zip::zip_list(file.path(getwd(), paste0(basefilename, ".zip"))) : 
  Cannot open zip file `/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip`
De plus : Message d'avis :
Dans normalizePath(zipfile) :
  path[1]="/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip": Aucun fichier ou dossier de ce type
Erreur dans withVisible(...elt(i)) : 
  Workflow failed during execution, check logs at: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/job-logs.txt

and log:

==================================================================================================] [32m650.19MB/s[0m, eta: [36m 0s[0m

[geoflow][zenodo][INFO] Validating entities 
[geoflow][zenodo][INFO] Validation issues --> 
  row        col    type                                               message
1   1 Identifier WARNING      Key is omitted, default key 'id' will be applied
2   1       Date WARNING       key 'edition' is not a recognized ISO date key.
3   1       Type WARNING Key is omitted, default key 'generic' will be applied
[geoflow][zenodo][INFO] Parsing 1 entities from tabular source 
[geoflow][zenodo][INFO] Parsing entity 1 
[geoflow][zenodo][INFO] Successfuly fetched 1 entities! 
[geoflow][zenodo][INFO] Enrich metadata entities from directory of contacts 
[geoflow][zenodo][INFO] 0009-0005-9789-9095 
[geoflow][zenodo][INFO] 0009-0005-9789-9095 
[geoflow][zenodo][INFO] 0000-0002-2592-8409 
[geoflow][zenodo][INFO] entropie-secretariat@ird.fr 
[geoflow][zenodo][INFO] julien.barde@ird.fr 
[geoflow][zenodo][INFO] 0009-0005-9789-9095 
[geoflow][zenodo][INFO] Successfuly loaded 1 entities! 
[geoflow][INFO] Check package dependencies for action 'zen4R-deposit-record' 
[geoflow][INFO] The following packages have been imported for action 'zen4R-deposit-record': 
  package installed version
1   zen4R      TRUE     0.9
[geoflow][zenodo][INFO] Creating 'zenodo' job directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/zenodo 
====================================================================================================
Workflow Execution
====================================================================================================
[geoflow][zenodo][INFO] Executing workflow job... 
----------------------------------------------------------------------------------------------------
[geoflow][zenodo][INFO] Executing software actions 'onstart' ... 
----------------------------------------------------------------------------------------------------
[geoflow][zenodo][INFO] Executing entity actions ... 
[geoflow][zenodo][INFO] Workflow mode: entity 
[geoflow][zenodo][INFO] Workflow with 1 actions 
[geoflow][zenodo][INFO] Action 1: zen4R-deposit-record 
[geoflow][zenodo][INFO] Create entity job dir at '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2' 
[geoflow][zenodo][INFO] Creating 'data' directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data 
[geoflow][zenodo][INFO] Creating 'metadata' directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/metadata 
[geoflow][zenodo][INFO] Entity working directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2 
[geoflow][zenodo][INFO] Copying data to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' 
[geoflow][zenodo][INFO] Copying data source 1 'session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip' (/home/sylvain/Documents/IRD/DATA3/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2.zip) to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230805080941/entities/session_2023_09_12_Surprise-DemoODM_Mavic2Pro_2/data' 
[geoflow][zenodo][INFO] Copying data to Job data directory from local file(s) 
[geoflow][zenodo][WARN] Source type: other 
[geoflow][zenodo][INFO] sourceZip = FALSE: source files will be uploaded 
[geoflow][zenodo][WARN] Metadata dynamic handling based on 'data' not implemented for source type 'other'

kikislater commented 1 year ago

Should I need to retest with uploadType:other ?

eblondel commented 1 year ago

I'm revising this issue. What is sure is that from now when when we have sourceType set to other , the following behaviors apply:

we don't unzip data, because we suppose it's unknown source type for which we don't want to process data to enrich the entity metadata.
we don't attempt to look into the zip files list (see my last commit), since we are doing this a re-zipped version of data files (essentially spatial data files)
Parameters such as sourceZip and sourceZipOnly were designed in the past, only for data files of type other. Since we don't want to unzip the content, these parameters are obsolete, and i've deprecated them

Based on this, you can reinstall and do your tests. The uploadType:other should not have effect here.

kikislater commented 1 year ago

Ok this explain some mysterious behaviours. So if I wanted to create new version then upload some files before publishing I ended with these actions:

    "actions": [
        {
            "id": "zen4R-deposit-record",
            "options": {
                "update_files": true,
                "communities": "uav",
                "depositWithFiles": false,
                "publish": false,
                "update_metadata": true,
                "strategy": "newversion",
                "zipEachDataFile": false,
                "deleteOldFiles": false
            },
            "run": true
        },
        {
            "id": "zen4R-deposit-record",
            "options": {
                "update_files": true,
                "communities": "uav",
                "depositWithFiles": true,
                "publish": false,
                "update_metadata": true,
                "strategy": "newversion",
                "zipEachDataFile": false,
                "deleteOldFiles": false
            },
            "run": true
        }        
    ]

eblondel commented 1 year ago

You don't need to declare twice the action. Being a generic uploader, this action will be triggered a second time at the end of the workflow, internally, without having it declared.

kikislater commented 1 year ago

Oops wrong discussion

kikislater commented 1 year ago

I was refering this : https://github.com/r-geoflow/geoflow/discussions/348 And particulary depositWithFiles

eblondel commented 1 year ago

yes i've realized :-) . The depositWithFiles as option should be decoupled from the internal parameter that is used in the workflow. Through the workflow, a generic uploader will be triggered a first time as defined in the workflow, and a second time at the end of the workflow, where files are actually deposited.

kikislater commented 1 year ago

Tested and got this but data in jobs is empty even if we got this from logs :

[geoflow][zenodo][INFO] Create entity job dir at '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230901162719/entities/uav_2023_05_24_ermitage_Mavic2Pro_02_1' 
[geoflow][zenodo][INFO] Creating 'data' directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230901162719/entities/uav_2023_05_24_ermitage_Mavic2Pro_02_1/data 
[geoflow][zenodo][INFO] Creating 'metadata' directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230901162719/entities/uav_2023_05_24_ermitage_Mavic2Pro_02_1/metadata 
[geoflow][zenodo][INFO] Entity working directory: /home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230901162719/entities/uav_2023_05_24_ermitage_Mavic2Pro_02_1 
[geoflow][zenodo][INFO] Copying data to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230901162719/entities/uav_2023_05_24_ermitage_Mavic2Pro_02_1/data' 
[geoflow][zenodo][INFO] Copying data source 1 'session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2.zip' (/home/sylvain/Documents/IRD/DATA2/session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2.zip) to entity job data directory '/home/sylvain/Documents/IRD/geoflow/geoflow-g2oi/jobs/20230901162719/entities/uav_2023_05_24_ermitage_Mavic2Pro_02_1/data' 
[geoflow][zenodo][INFO] Copying data to Job data directory from local file(s)

eblondel commented 1 year ago

Can you run the below code:

datasource_uri = "/home/sylvain/Documents/IRD/DATA2/session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2.zip"
data.files <- list.files(path = dirname(datasource_uri), pattern = basename(datasource_uri))

and tell me if you get a result for data.files

kikislater commented 1 year ago

Yes of course file exists, I doubled check:

> datasource_uri = "/home/sylvain/Documents/IRD/DATA2/session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2.zip" 
> data.files <- list.files(path = dirname(datasource_uri), pattern = basename(datasource_uri)) 
> data.files 
[1] "session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2.zip"

And in data I have this

source:session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2.zip@/home/sylvain/Documents/IRD/DATA2/session_2022_09_12_mayotte-fmr-surprise_Mavic2Pro_2.zip_
sourceType:other_
sourceZipOnly :false_
upload:true

eblondel commented 1 year ago

Fixed with https://github.com/r-geoflow/geoflow/commit/16e31c6d6624b0b3256cec1ac5080bc4be915dab

kikislater commented 1 year ago

:heavy_check_mark: It works ! 0.9 version of zen4R minimum

r-geoflow / geoflow

Prevent default data unzipping when sourceType is 'other' #344

==================================================================================================== Workflow Execution

[geoflow][zenodo][INFO] Executing workflow job...

[geoflow][zenodo][INFO] Executing software actions 'onstart' ...