issues grass7_2018

[tpilz]

- to be continued -

Remaining issues I observed with my test dataset along with GRASS 7 adaptation:

• [x] calc_subbas(): message of number of subbasins left after removing spurious subbasins is wrong (i.e. not the actual number found in GRASS) -> resolved by commit 3fc4c672dc61e7f83c5bcc869ce121c57008a364
• [x] calc_subbas(): differing number of categories in subbas and drainage points might be intentional (e.g. when drain_points and thresh_sub are given) -> remove or adjust warning message -> resolved by commit 3fc4c672dc61e7f83c5bcc869ce121c57008a364
• [x] calc_subbas(): column subbas_id in output <points_processed>_snap does not correspond to categories in output basin_out -> resolved by commit 3fc4c672dc61e7f83c5bcc869ce121c57008a364
• [x] calc_subbas(): if column subbas_id is given in input drain_points the categories in the resulting basin_out raster do NOT correspond to this column as promised by the documentation -> resolved by commit 3fc4c672dc61e7f83c5bcc869ce121c57008a364
• [x] area2catena(): GRASS reclass files produced even with argument grass_files = FALSE; this occurs only if an element in arg supp_qual comes without explicit mapset declaration
• [x] reservoir_lumped() and reservoir_strategic(): @TillF Adaptation to Windows might be necessary (relates to readVECT() mapset issues)
• [x] db_create(): When trying to re-create an existing database v. 26 with overwrite="drop" there is an error: Table 'x_seasons' already exists when updating to version 26. Rename / delete manually, and repeat update. Note: The message with x_seasons is exemplary, when removing x_seasons the same error occurs with the next table. I don't understand how I should overwrite an existing table.
• [x] db_create(): does not work with overwrite = 'empty' applied to an existing database (error raised by internal call to db_update(), see post below.

Observed changes in behaviour in comparison to lumpR v2.5.0, latest GRASS 6.4 based version (@TillF check if this could be reasonable):

• [x] order of subbasins (i.e. their ID) in output files changed (but statistics, i.e. cell counts for specific subbasins, remain unchanged) -> intended and no problem
• [x] lump_grass_prep(): differences in the EHA map (IDs -> no problem; slightly different EHA sizes, max. 13 grid cells in my test setup) -> _I made some tests and it turned out the raw outputs of r.watershed (i.e. argument half.basin/half_basin) are different, even with the same input data, threshold values and the same algorithm (SFD)_
• [ ] results from prof_class() deviate slightly in some occasions (TC definition), even with the same seed and the same input files and parameter settings (should not be problematic)
• [x] reservoir_outlet(): for vector map res_vct columns res_id and name are now required. I don't understand the reason, I think this is unnecessary. Maybe an argument should be added where the user can chose which column contains reservoir IDs (the standard should be column cat). -> _removed necessity to contain column name; res_id should be fine_
• [ ] reservoir_lumped(): When looking at the source code I get the feeling that there are superfluous commands resulting in higher computation time (e.g. res_lump <- readVECT(res_vect_class, type="point") in lines 323 and 341

Function execution times:

• [x] calc_subbas() now takes twice as much execution time in my example -> seems to be case specific, in a further test I made with different data it was faster... I guess, on the one hand GRASS operations are faster, but at the other hand some recent extensions (specifically, ensuring that drain points are not directly in the middle of a cell which includes some additional GRASS as well as read and write operations) make it slower
• [x] lump_grass_prep() takes less than half of the former runtime which is good but might be connected to the issue above? -> GRASS operations are faster, different results caused by different behaviour of r.watershed, see above
• [ ] reservoir_lumped() is considerably slower

- to be continued -

[TillF]

I'll respond point by point: area2catena(): I could not reproduce this. Reclass files are produced as soon as an svc-layer is present, regardless of the flag. Does this explain your observation?

[TillF]

reservoir_lumped(): Tried to test this, but it requires reservoir_outlet() to be run first. This routine has not yet migrated to grass7. I started off, but I guess you can do this quicker.

[TillF]

• -order of subbasins: Probably a result of your workaround with R-categories? Who cares, as long as the results stay the same.

• results from prof_class() differ: The clustering includes random components, which may respond to some of the changes I implemented. Not worth to trace this back, I guess.

• EHA IDs changed: In eha-raster? compared to GRASS6? This would not be problematic, would it?

• results of area2catena() differ: I did not touch the actual computations, I cannot explain this.

[tpilz]

area2catena(): I could not reproduce this. Reclass files are produced as soon as an svc-layer is present, regardless of the flag. Does this explain your observation?

In my case the reclass files are only produced if svc is given without explicit mapset reference via "@" operator. Furthermore, I don't understand why these files shall be produced when grass_files = FALSE.

order of subbasins: Probably a result of your workaround with R-categories? Who cares, as long as the results stay the same.

I agree.

EHA IDs changed: In eha-raster? compared to GRASS6? This would not be problematic, would it?

It wouldn't, I just wondered. But it's not important.

results of area2catena() differ: I did not touch the actual computations, I cannot explain this

For some EHAs I also have slightly different cell counts than before. I only cannot identify the reason for this.

[tpilz]

Note: the grass_files issue of area2catena() was introduced in commit afbc509c07a12a1af583ce4c24630196b1ecb853 but still I don't get the point.

[tpilz]

@TillF see latest updates of remaining issues in the initial post.

[TillF]

From my side, lumpR for GRASS7 can be released.

[tpilz]

db_create() now works with overwrite = 'drop' but not with overwrite = 'empty:

Error in SQL query execution while updating db.
Query:   ALTER TABLE vegetation ADD COLUMN intfc DOUBLE ;
error-message: HY000 1 [SQLite]duplicate column name: intfc (1)

which seems to be raised by an internal call to db_update(dbname, db_ver).

[A-Mue]

Tested on Windows, for an approx. 80 000 km² catchment (30m-resolution DEM):

A) Execution from apply_lumpR_template calc_subbas() Problem: nothing happens, but no error messages

Issued meassages:

_%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % START calcsubbas() % Initialise function... ... % Calculate drainage and river network... ... ABSCHNITT 4: Schließe Karten. % OK

--> duration ca. 6 min

_% Snap given drainage points to streams... Exporting 116 features... VOUTOG1 komplett. 116 features (Point type) written to (SQLite format). Exporting 24786 features... VOUTOG1 komplett. 24786 features (Line String type) written to

(SQLite format). % OK_ **--> duration ca. 10 min** _% Calculate catchments for every drainage point..._ --> This is the last issued message, but nothing happens anymore (run over night) --> Output created: basin_calc_t (spurious subbas not yet eliminated, map not clipped) basin_recl basin_outlet_t **B) Execution with local version of "subbasin.R" (attached)** **1. Problem with deletion of temporary Grass files** The temporary files created by GrassGIS should be deleted several times in "subbasin.R", but R issued the message "Permission denied". _> if(.Platform$OS.type == "windows") { dir_del <- dirname(execGRASS("g.tempfile", pid=1, intern=TRUE, ignore.stderr=T)) files_del <- grep(substr("dp_t", 1, 8), dir(dir_del), value = T) file.remove(paste(dir_del, files_del, sep="/")) }_ I manually deleted the files in the folder ".../GrassGIS_Data/Location/Mapset/.tmp/unknown/" **2. Subbasin at catchment outlet is set to NA** Line-by-line exection of "subbasin.R", lines skipped: 366–430 In map "basin_all_t": last subbas at outlet has ID 0; this command creates NA: _> execGRASS("r.null", map=basin_out, setnull="0")_ In the next step, this warning is issued: _> no_cross <- length(execGRASS("r.stats", input=basin_out, flags=c("n"), intern=T, ignore.stderr = T)) > if(no_catch != no_cross) warning... Warning message: Number of categories in subbas not equal to number of drainage points! This might be because ..._ I "quick-fixed" this, by manually reclassifying "basin_all_t" (0=1, 1=2, ..., 115=116) and named this new map "subbas". [subbasin.txt](https://github.com/tpilz/lumpR/files/2525767/subbasin.txt)