search

ARGANS / shellfish_and_algae-MODEL

0 stars 0 forks source link

Unable to integrate code from advectionPrototype directory #4

Open nmaltsev opened 2 years ago

nmaltsev commented 2 years ago

The code doesn't work and I don't understand what values need to be entered to execute the code:

  1. Unknown properties. Looks like Maël's code needs more properties than the form provides. He couldn't explain to me how can I get the dateBeginning and dateEnd properties:

    • # sortedList = sortData(parms_run['dateBeginning'], parms_run['dateEnd'], len(dataNO3))
    • code

  2. I also have a question about the file_name property used in the code in this link https://github.com/ARGANS/shellfish_and_algae-MODEL/blob/main/scenario_b/start.py#L44-L46 I used /media/global/PAR_ref_OC_2020.nc as the value to resume simulation without exception. Do you know what the value should be?

qjutard commented 2 years ago
  1. I believe you only need the 'dateBeginning' for the code to work ? The date is the first day of the month in 'deployment_month' (number 1-12) of the year in 'year'
  2. This if fine for now, in the future this will be the file used only when "NASA Ocean Colour 2020, filled" is the data selected for PAR (TODO in the pretreatment phase). But right now this is the only working file.
qjutard commented 2 years ago

I added Mael to the ticket in case you need further help to solve this

nmaltsev commented 2 years ago

I pushed my update to the repository. I don't know how to make the code work. I am blocked by this exception.

sys:1: FutureWarning: In a future version of pandas all arguments of read_csv except for the argument 'filepath_or_buffer' will be keyword-only
Traceback (most recent call last):
  File "/opt/start.py", line 103, in <module>
    latRef = np.ones((np.shape(dataEwc[0])[1], np.shape(dataEwc[0])[0])) * latitudes[latitudeMin:latitudeMax]
  File "/usr/local/lib/python3.10/site-packages/numpy/ma/core.py", line 4179, in __rmul__
    return multiply(other, self)
  File "/usr/local/lib/python3.10/site-packages/numpy/ma/core.py", line 1015, in __call__
    result = self.f(da, db, *args, **kwargs)
ValueError: operands could not be broadcast together with shapes (36,18) (17,)
ghost commented 2 years ago

Hello,

Do you put as inputs some coordinates? Maybe when it imports the data it doesn't cut the data array in same place, normally the number of rows in dataEwc have to be equals to the length of latitudes. I don't know why algaeData.parameterData['eastward_Water_current'].getVariable(sim_area)[0] don't return something with the same row length as algaeData.parameterData['Temperature'].getVariable('latitude',sim_area). Normally Temperature and eastward_Water_current are on the same grid. Maybe try to replace dataEwc by dataTemp in line 103, but I think it will maybe just change place to the issue because it seems that the data don't have all the same shape.

Maël

De : Nikolai MALTSEV @.***> Envoyé : mardi 16 août 2022 15:56:07 À : ARGANS/shellfish_and_algae-MODEL Cc : Maël Jaouen; Assign Objet : Re: [ARGANS/shellfish_and_algae-MODEL] Unable to integrate code from advectionPrototype directory (Issue #4)

I pushed my update to the repository. I don't know how to make the code work. I am blocked by this exception.

sys:1: FutureWarning: In a future version of pandas all arguments of read_csv except for the argument 'filepath_or_buffer' will be keyword-only Traceback (most recent call last): File "/opt/start.py", line 103, in latRef = np.ones((np.shape(dataEwc[0])[1], np.shape(dataEwc[0])[0])) latitudes[latitudeMin:latitudeMax] File "/usr/local/lib/python3.10/site-packages/numpy/ma/core.py", line 4179, in rmul return multiply(other, self) File "/usr/local/lib/python3.10/site-packages/numpy/ma/core.py", line 1015, in call result = self.f(da, db, args, **kwargs) ValueError: operands could not be broadcast together with shapes (36,18) (17,)

— Reply to this email directly, view it on GitHubhttps://github.com/ARGANS/shellfish_and_algae-MODEL/issues/4#issuecomment-1216673546, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXKYYH5UURYGYLMTN67IYODVZOMXPANCNFSM55TEYI2A. You are receiving this because you were assigned.Message ID: @.***>

nmaltsev commented 2 years ago

My observations:

  1. I still have the same problem if I use bounded area coordinates: parms_run['min_lon'] = -4 parms_run['max_lon'] = -3 parms_run['min_lat'] = 48.5 parms_run['max_lat'] = 49

  2. In case I use the default scope range (180..-180, 90..-90) I get memory inflation and system reboot due to my VM crashes. I did my best but the code is not working.

qjutard commented 2 years ago

About point 2: Could you compare how much RAM is available on the VM versus on the machine Mael used for prototyping ?

ghost commented 2 years ago

On the computer used for prototyping I have 16 Go RAM

De : qjutard @.***> Envoyé : jeudi 18 août 2022 10:20:26 À : ARGANS/shellfish_and_algae-MODEL Cc : Maël Jaouen; Assign Objet : Re: [ARGANS/shellfish_and_algae-MODEL] Unable to integrate code from advectionPrototype directory (Issue #4)

About point 2: Could you compare how much RAM is available on the VM versus on the machine Mael used for prototyping ?

— Reply to this email directly, view it on GitHubhttps://github.com/ARGANS/shellfish_and_algae-MODEL/issues/4#issuecomment-1219179021, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXKYYHZJG6IMQIU7PEB47P3VZXW4VANCNFSM55TEYI2A. You are receiving this because you were assigned.Message ID: @.***>

nmaltsev commented 2 years ago

I tested the code on a virtual machine with 12 GB of RAM + 12 GB of disk swap

nmaltsev commented 2 years ago

Our production server at AdwiseEO has 31.3GB of RAM with 1.6GB of disk swap. This is a huge amount, close to the physical limits of a bare metal server, but all this memory will be used for other parallel executions.

I have been testing the code for a long time. And it doesn't seem to work properly. I'm still not sure about the inputs. I used the results I got from the pretreatment scripts. And I used the dateBegining and dateEnd properties from the dataimport task.

ghost commented 2 years ago

The issues still came from

latRef = np.ones((np.shape(dataEwc[0])[1], np.shape(dataEwc[0])[0])) * latitudes[latitudeMin:latitudeMax]

?

If you want we can organize a meeting to see where does the issue comes from.

You can try to put (None,None,None,None) as default value, the issue maybe comes from the fact that (180..-180, 90..-90) is bigger than the data bounds.

De : Nikolai MALTSEV @.***> Envoyé : jeudi 18 août 2022 11:29:08 À : ARGANS/shellfish_and_algae-MODEL Cc : Maël Jaouen; Assign Objet : Re: [ARGANS/shellfish_and_algae-MODEL] Unable to integrate code from advectionPrototype directory (Issue #4)

Our production server at AdwiseEO has 31.3GB of RAM with 1.6GB of disk swap. This is a huge amount, close to the physical limits of a bare metal server, but all this memory will be used for other parallel executions.

I have been testing the code for a long time. And it doesn't seem to work properly. I'm still not sure about the inputs. I used the results I got from the pretreatment scripts. And I used the dateBegining and dateEnd properties from the dataimport task.

— Reply to this email directly, view it on GitHubhttps://github.com/ARGANS/shellfish_and_algae-MODEL/issues/4#issuecomment-1219250731, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AXKYYH3T7WVQURBLKKC4WALVZX66JANCNFSM55TEYI2A. You are receiving this because you were assigned.Message ID: @.***>

qjutard commented 2 years ago

I don't expect that using None will change anything, using larger bounds than the data on getVariable() calls just gets all of the range and nothing more.