search

Earth-Information-System / fireatlas

3 stars 1 forks source link

Run on VEDA #50

Closed jsignell closed 1 month ago

jsignell commented 1 month ago

When I was running California 2023 for V3 validation I noticed that Fire_Forward for that whole year took about 4 hours on MAAP ADE and about 1 hour on VEDA (with 7.4 GB RAM and 3.7 CPU)

These are the breakdowns:

month VEDA ADE
Jan 00:01:39 00:09:10
Feb 00:04:05 00:24:00
Mar 00:03:19 00:18:18
Apr 00:02:29 00:12:37
May 00:02:02 00:09:35
Jun 00:02:19 00:10:55
Jul 00:02:51 00:12:55
Aug 00:07:11 00:28:57
Sep 00:14:01 00:41:30
Oct 00:11:28 00:27:31
Nov 00:09:13 00:25:28
Dec 00:07:47 00:18:26

VEDA can't write to MAAP buckets but it can read from them, so there is no point in moving the input files, but it would be interesting enable writing preprocess and output files to a different bucket. This should be a relatively straightforward change to FireConsts.

jsignell commented 1 month ago

Greg had the idea that we could just copy the ingest data over and then if you change the S3_PATH everything just works. See #56 for run notebooks