Why predict FWI from BUI and ISI? Their relationship is known analytically

[cvitolo]

@eduardogfma and @mariajoaosousa thanks for uploading the first modelling notebook. It makes the review much easier.

Background: you got from here 3 indices of fire danger: FWI, ISI and BUI. As mentioned before, the relationship FWI = f(ISI, BUI) is known analytically (equations are here: http://cfs.nrcan.gc.ca/pubwarehouse/pdfs/19973.pdf). Therefore, FWI/ISI/BUI are not independent from one another. FWI is calculated from ISI and BUI.

Question: I see you are setting up a model to predict FWI from ISI and BUI. Can you explain how learning this same relationship from a model can be useful in operation/practical terms?

[eduardogfma]

It is true FWI is known analytically and depends on BUI and ISI. However, if I understand it correctly, FWI is computed for a given geographical location, i.e. the analytical computations disregard the spatial dependencies between points. More, as you mentioned earlier the analytical approach fails everywhere except Canada and Russia. Thus, if we want to get some additional knowledge, the analytical approach is not a viable option.

The uploaded notebook was not meant to be an operational/practical tool, we were only interested in testing and presenting the available models. Up until that point we assumed the proposed machine learning techniques (NN and SVM) would be able to capture the system's behaviour. With this notebook it was shown that (1) the SVM framework is able to model that non-linear relationship, and (2) how good a performance can be achieved by only taking in consideration the spatial relationship between geographical locations. This last point is important, because (1) it strengthens our initial assumption that there is a temporal component that can not be disregarded (which may have implications in the future chosen models, either in the SVM or NN domains), and (2) it raises new questions and, as a result, new research avenues.

From here, knowing that the framework works reasonably well, we can input other features (e.g. radiative power, burned area, etc.) and see which ones may be of interest for the FWI estimation (in and outside Canada). Ultimately, what we want is to have a generic tool that works all around the globe.

We understand that everyone involved expects results as quick as possible, but actually I do not think we are behind schedule. This is an exploratory project, no one seems to have answers to the problem we are trying to solve. What we know is that people outside Canada are able to build models around the analytically computed FWI, which indicates that there must be a way of automatizing, generalizing and improving those estimations.

[cvitolo]

I think it would make sense to predict FWI from BUI and ISI if you were trying to build a statistical emulator for GEFF (the model ECMWF uses to generate gridded fire indices, e.g. FWI, BUI, ISI), however this is not what you are doing/claiming. Am I right? I think it's going to be clearer if we discuss your statements point-by-point below. If the conversation becomes confusing, we could split the topics in different issues to investigate them separately.

It is true FWI is known analytically and depends on BUI and ISI. However, if I understand it correctly, FWI is computed for a given geographical location, i.e. the analytical computations disregard the spatial dependencies between points.

I confirm the FWI is calculated for each grid cell independently, therefore the FWI (provided by ECMWF) is not supposed to explain spatial dependencies. Therefore, if your model is emulating FWI, it should not explain spatial dependencies. If your model is not emulating FWI, but doing something else ... please elaborate on how these spatial dependencies are explored.

More, as you mentioned earlier the analytical approach fails everywhere except Canada and Russia.

This is not quite what I said, there are places where FWI estimates have smaller biases and have been positively validated by local agencies, e.g many countries in Europe, Canada, etc. There are other places where large biases have been observed (e.g. Arctic, Indian and Pacific oceans... see this article for more info).

Thus, if we want to get some additional knowledge, the analytical approach is not a viable option.

I don't understand, can you elaborate more on this please?

The uploaded notebook was not meant to be an operational/practical tool, we were only interested in testing and presenting the available models.

Of course, I was more interested in knowing how you think ECMWF could make use of the results of your project.

Up until that point we assumed the proposed machine learning techniques (NN and SVM) would be able to capture the system's behaviour. With this notebook it was shown that (1) the SVM framework is able to model that non-linear relationship, and (2) how good a performance can be achieved by only taking in consideration the spatial relationship between geographical locations. This last point is important, because (1) it strengthens our initial assumption that there is a temporal component that can not be disregarded (which may have implications in the future chosen models, either in the SVM or NN domains), and (2) it raises new questions and, as a result, new research avenues.

Can you explain how your model explains the spatial relationship between geographical locations? I don't see that explained in your notebook.

From here, knowing that the framework works reasonably well, we can input other features (e.g. radiative power, burned area, etc.) and see which ones may be of interest for the FWI estimation (in and outside Canada). Ultimately, what we want is to have a generic tool that works all around the globe.

Your model emulates FWI reasonably well, but again I'm not sure if this is actually your goal. Also FWI quantifies a potential danger, while fire radiative power and burned areas quantify a real/observable danger. Can you explain how fire radiative power and burned areas can help you in this context?

We understand that everyone involved expects results as quick as possible, but actually I do not think we are behind schedule.

I'm not commenting on the timing, but on the methodology.

This is an exploratory project, no one seems to have answers to the problem we are trying to solve. What we know is that people outside Canada are able to build models around the analytically computed FWI, which indicates that there must be a way of automatizing, generalizing and improving those estimations.

Your model, as it is, cannot improve FWI estimations but only replicate/emulate them. If you want to improve estimations, you should train your model against 'observed FWI' (fire radiative power and burned areas could be useful in this context) not the one calculated from BUI and ISI.

[eduardogfma]

This is not quite what I said, there are places where FWI estimates have smaller biases and have been positively validated by local agencies, e.g many countries in Europe. There are other places where large biases have been observed

This is what I meant by failing, although you may actually make a clear difference between having a bias and completely failing to predict the value at all sampling times. I am fine with that approach. We are interest precisely on that bias you mentioned. Namely, we want to autonomously determine those from data.

Thus, if we want to get some additional knowledge, the analytical approach is not a viable option.

My point here was the following. If we both agree that the FWI is incorrect (either by presenting a bias, or simply because it is impossible to get the predictions right) in some locations, than the analytical approach can only provide useful information if we know the biases a priori. That is, the analytical approach is unable to provide any indication regarding those biases, nor which other features may be of value to improve the FWI estimation. In other words, the analytical approach is a static one. In turn, the machine learning (ML) approaches we proposed can (1) implicitly capture those biases across the region of interest (ROI), and (2) dynamically/iteratively improve its performance over time. By no means I am advocating for a complete substitution of approaches, I am simply claiming having those two approaches in parallel can greatly improve both the analyses and predictions.

Of course, I was more interested in knowing how you think ECMWF could make use of the results of your project.

Our motivation was always to provide introductory notebooks to interested readers in ML techniques applied to wild fire events, this is why we based our approach on the knowledge discovery in databases framework, as written in the submitted proposal.

Can you explain how your model explains the spatial relationship between geographical locations? I don't see that explained in your notebook.

In support vector machine (SVM) regression, the basic idea is to determine the hyperplane which maximizes the margin, where margin is a distance between the hyperplane and a given point. The key point is that some degree of error tolerance is employed, meaning that a given point, $i$, will be affected in some degree by the surrounding points ($i-1$ and $i+1$, if we consider a 2D space). In general we are working in multi-dimensional space, so there will exist a cloud of points exerting influence over point $i$ at all times. Moreover, a similar effect is expected when using neural networks (NN), particularly due to the convolution operations, where the model is capable of extracting multiple dependencies at multiple levels -- this is why people continue using convolution NN in computer vision.

Your model emulates FWI reasonably well, but again I'm not sure if this is actually your goal. Also FWI quantifies a potential danger, while fire radiative power and burned areas quantify a real/observable danger. Can you explain how fire radiative power and burned areas can help you in this context?

It seems you are always considering the same line of thought:

big fire => high radiative power + long burned areas => big severity => high FWI.

There is nothing wrong with this line of thought, everyone agrees with it. However, this does not provide any information about the locations in which a high FWI was determined but no fire had occurred, i.e. no one can guarantee that the model you used (analytical or not) resulted in a good prediction. A prediction is a prediction, and until the event actually takes place nothing is validated. Thus, why not take the problem backwards, i.e.:

high FWI ~> big severity ~> high radiative power + long burned areas,

where '~>' was meant to symbolize 'indicates'. Note that in this case we end up exactly in the same position, until a fire occurs no one can know how good a prediction it was. However we can investigate the relationships between several features and determine their relative importance across a given ROI over time to try to determine the biases that may exist in different locations. To summarize, it seems to us that your take on this, although correct, kills the possibility of further investigation, since it seems to be contradictory: on the one hand, the analytical approach is a good one because it provides good FWI estimates when fires actually occur, on the other hand if a fire didn't take place the FWI prediction is not correct, although the same estimation process has been employed.

Your model, as it is, cannot improve FWI estimations but only replicate/emulate them

In Canada, of course the estimations will not be improved. The point is to improve outside Canada, where estimations fail (either because of biases or simply wrong predictions). We use Canada as a ground truth, which implies that our assumption is that there exists a nominal FWI scale, i.e. that an FWI value of 30 in Canda is equivalent to a FWI value of 30 in Portugal.

If you want to improve estimations, you should train your model against 'observed FWI' (fire radiative power and burned areas could be useful in this context) not the one calculated from BUI and ISI.

We have been using the recommended datasets. If there is more adequate ones (i.e. 'observed FWI'), please feel free to share. However, the methodology and models previously developed and employed will suffer no change, only the inputs will be different.

[cvitolo]

I think there are too many things mentioned here, it's probably better to focus on one thing at the time.

The main point is that when you talk about bias you mean the difference between the GEFF-FWI (nc file you got from Zenodo, GEFF is the model used to generate fire danger indices) and the FWI your model predicts from ISI and BUI. Your current approach won't generate an improved FWI, compared to GEFF. The best it can do is to make the same prediction as GEFF. Do you agree?

[eduardogfma]

The main point is that when you talk about bias you mean the difference between the GEFF-FWI (nc file you got from Zenodo, GEFF is the model used to generate fire danger indices) and the FWI your model predicts from ISI and BUI.

No, when I talk about bias I mean there is a mismatch between the FWI values computed by the GEFF-FWI and the actual/true FWI values that may be estimated after a fire incident takes place. This is in general true for any region, but is particular important (in our view) for locations outside Canada. This bias mitigation is what we have been referring to as calibration.

Your current approach won't generate an improved FWI, compared to GEFF. The best it can do is to make the same prediction as GEFF.

Our current work does not yet produce better estimates than GEFF. Up until now, we have been focused on building an infrastructure that is able to receive as many inputs/features as needed. And to validate that this infrastructure is working we need some comparison, in this case the Canadian FWI. Once we can achieve as good as GEFF's estimations, then we can input more features, e.g. radiative power and burned areas. It is our belief that those features are highly correlated with FWI, and by introducing them in the models we will be able to capture the degree each one of these actually has over the final FWI estimation.

A second step would be to analyse how this trained models actually behave in other regions. Here it is important to mention again that our fundamental assumption is that there is a nominal FWI scale. Assuming that is true, then pre-training the models with the Canadian FWI will enable them to capture the fundamental relationships between variables, and then by training once again using the local features the model will adapt those captured fundamental relationships to the local conditions, or so we hope.

[cvitolo]

Probably there is a misunderstanding, let's look at this in a different way.

Let's assume after your first experiment with Canada (where GEFF-FWI is reasonably close to the an 'hypothetical true value'), you decide to train your model on Indonesia. In Indonesia, GEFF-FWI is not as reliable as in Canada. How can you make better estimates than GEFF-FWI? And how are you going to measure the improvement?