OPM / opm-simulators

Simulator programs and utilities for automatic differentiation.
http://www.opm-project.org
GNU General Public License v3.0
110 stars 120 forks source link

THERMAL model RTEMPI error #3969

Open EdmundStephens opened 2 years ago

EdmundStephens commented 2 years ago

[v2022.10] We are running some shallow geothermal evaluations, for which we have a layer subcropping to the sea bed and a geothermal gradient from 40 F (sea water) down to about 80 F at 600m depth. I am trying to use the TEMPI initial temperature to initialise the model, but this is triggering some sort of difficulty ...

Attached example runs:

When TEMPI is not a constant, the solver backup up in the first timestep:

Error: Solver failed to converge after cutting timestep 10 times.
Program threw an exception: [/build/opm-simulators-9Blz1A/opm-simulators-2021.10-rfinal/opm/simulators/timestepping/AdaptiveTimeSteppingEbos.hpp:543] Solver failed to converge after cutting timestep 10 times.
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 16 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[ABZ50103WS:30443] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193
[ABZ50103WS:30443] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193
[ABZ50103WS:30443] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193
[ABZ50103WS:30443] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193
[ABZ50103WS:30443] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193
[ABZ50103WS:30443] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193
[ABZ50103WS:30443] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193
[ABZ50103WS:30443] PMIX ERROR: UNREACHABLE in file ../../../src/server/pmix_server.c at line 2193

Unclear what is the issue here ... might be something in the input, or something in Flow. I'd be very grateful if you could have a check in the debugger. Unfortunately in this case it is not possible to check in Eclipse as TEMPI is an E300-only keyword.

TEMPI1.zip TEMPI2.zip TEMPI3.zip

atgeirr commented 2 years ago

The convergence stalls, with a pattern like this:

Iter    MB(O)      MB(W)      MB(G)      MB(E)      CNV(O)     CNV(W)     CNV(G)     CNV(E) 
   0  0.000e+00  2.726e-19  0.000e+00  2.312e-18  0.000e+00  1.337e+01  0.000e+00  1.910e+02
   1  0.000e+00  2.650e-09  0.000e+00  1.180e-01  0.000e+00  3.131e-02  0.000e+00  5.998e-01
   2  0.000e+00  2.909e-09  0.000e+00  1.180e-01  0.000e+00  1.472e-03  0.000e+00  1.182e-01
   3  0.000e+00  3.228e-09  0.000e+00  1.180e-01  0.000e+00  8.189e-04  0.000e+00  1.182e-01
   4  0.000e+00  3.457e-09  0.000e+00  1.180e-01  0.000e+00  5.205e-04  0.000e+00  1.182e-01

... no matter how much the timestep is cut. It could be that we have some error in scaling of the energy residual (have not looked at this in detail now), and that this is actually ok. Probably not, but you could test by increasing the CNV tolerance to something larger than 1.182e-1 and see if the solution seems ok? (That would be a far too coarse CNV tolerance for the fluids though. So only recommending this as a test.)

EdmundStephens commented 2 years ago

Various solver settings tried to no effect ... shows the same time step cutting.

tolerance-cnv=0.3
tolerance-cnv-relaxed=1
tolerance-mb=1e-04 to 1e-06 tried
tolerance-well-control=1e-05 to 1e-07 tried
tolerance-wells=1e-02  to 1e-04 tried
OPMUSER commented 2 years ago

My two cents:

There is a discontinuity in TEMP which is bound to cause numerical issues for the TEMP1 case as shown below: image Also the grid cells are non-orthogonal, which is never a good idea. image TEMP2 is more reasonable as there is a temperature gradient, no temperature discontinuity. image

Looking at the PORV,

image

So I would expect that the grid is the issue here, but that does not eliminate a numeric issues at this stage.

Can you re-try using a more conventional grid and see if you still have the same issue?

For setting the temperature versus depth one can try using the RTEMPTVD keyword.

atgeirr commented 2 years ago

The most suspicious thing (to me) about the grid is the enormous difference in porevolumes between the top part (representing the sea I think) and the rest, and the huge transmissibilities in the sea part. That may lead to problems we have not thought about before. Could you try to eliminate the extra pore volume and huge transmissibilites for that part, and see if it at least makes it run through?

Edit: To be clear, we already deal pretty well with numerical aquifers where you also treat some cells as having huge pore volume, so it is something we should be able to handle.

EdmundStephens commented 2 years ago

Thanks for the excellent feedback, and indeed the upper layers are representing the connection on the sea bed (rather optimistically in this case) to the outcropping unit.

Attached a version with TEMPI4 with upperlayers perm=1000 md and no PV mult ... same problem. TEMPI4.zip

EdmundStephens commented 2 years ago

Again thank you and to address David's suggestions. The following models using the RTEMPVD make a much nicer bug test, where the FLOW run backs up at the first time step and the equivalent E100 runs works: TEMPI4_RTEMPVD_ECL.zip TEMPI4_RTEMPVD_FLOW.zip

The E100 run is identical except for THERMAL-->TEMP and the THCWATER/OIL/GAS removed (these are E300 only for some reason).

I also ran a case without the unconformity to check out if the grid connections are triggering the issue. The case has blanked out layers 1 - 5 and has the same convergence issue backing up at the first time step. TEMPI4_RTEMPVD_FLOW1.zip

OPMUSER commented 2 years ago

@atgeirr Is this the issue https://github.com/OPM/opm-common/pull/3068 ?

Simulation would not run with TEMP keyword for water single-phase case.

atgeirr commented 2 years ago

@atgeirr Is this the issue OPM/opm-common#3068 ?

Simulation would not run with TEMP keyword for water single-phase case.

Not sure, but I do not think so?

OPMUSER commented 2 years ago

Not sure, but I do not think so?

@atgeirr Yep you are right, both the TEMPI4_RTEMPVD_FLOW and TEMPI4_RTEMPVD_FLOW1 case have the same problem with the current master build. I even ran without the THCWATER/OIL/GAS keywords - same problem.

There is only water in these models, but the declared phases include oil, gas etc., but that should not be a problem as the E100 case runs without issue. I have check both the E100 and the equivalent OPM Flow model and the initialization is identical. Thus, I think this is a numeric issue and a bug.

Don't know what else to suggest, as this is a bug.

goncalvesmachadoc commented 2 years ago

There is only water in these models, but the declared phases include oil, gas etc., but that should not be a problem as the E100 case runs without issue. I have check both the E100 and the equivalent OPM Flow model and the initialization is identical. Thus, I think this is a numeric issue and a bug.

Have you tried to run the same case with only water phase declared?

OPMUSER commented 2 years ago

@goncalvesmachadoc

Yes got this:

System:  Single phase Water system. Nothing to check
Program threw an exception: Could not initialize the problem: VISCREF is required when WATVISCT is present

And when I add back in VISCREF I get my favorate OPM Flow message:

Report step  0/13 at day 0/366, date = 01-Jan-2020
flow: /home/david/OPM/GitHub/opm-material/opm/material/fluidsystems/BlackOilFluidSystem.hpp:1532: static short int Opm::BlackOilFluidSystem<Scalar, IndexTraits>::canonicalToActivePhaseIdx(unsigned int) [with Scalar = double; IndexTraits = Opm::BlackOilDefaultIndexTraits]: Assertion `phaseIsActive(phaseIdx)' failed.
[david-VirtualBox:02186] *** Process received signal ***
[david-VirtualBox:02186] Signal: Aborted (6)
[david-VirtualBox:02186] Signal code:  (-6)
[david-VirtualBox:02186] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f38b2441520]
[david-VirtualBox:02186] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7f38b2495a7c]
[david-VirtualBox:02186] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7f38b2441476]
[david-VirtualBox:02186] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7f38b24277f3]
[david-VirtualBox:02186] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x2871b)[0x7f38b242771b]
[david-VirtualBox:02186] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7f38b2438e96]
[david-VirtualBox:02186] [ 6] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x15e0211)[0x55e2bdd51211]
[david-VirtualBox:02186] [ 7] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x165808b)[0x55e2bddc908b]
[david-VirtualBox:02186] [ 8] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x166bfb4)[0x55e2bdddcfb4]
[david-VirtualBox:02186] [ 9] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x166c3c6)[0x55e2bdddd3c6]
[david-VirtualBox:02186] [10] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x169515e)[0x55e2bde0615e]
[david-VirtualBox:02186] [11] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x1695b4b)[0x55e2bde06b4b]
[david-VirtualBox:02186] [12] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x16861de)[0x55e2bddf71de]
[david-VirtualBox:02186] [13] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x15f60a1)[0x55e2bdd670a1]
[david-VirtualBox:02186] [14] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x235cb5)[0x55e2bc9a6cb5]
[david-VirtualBox:02186] [15] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f38b2428d90]
[david-VirtualBox:02186] [16] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f38b2428e40]
[david-VirtualBox:02186] [17] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x34a7d5)[0x55e2bcabb7d5]
[david-VirtualBox:02186] *** End of error message ***

Deck attached. TEMPI4_RTEMPVD_FLOW3-DBx.zip

goncalvesmachadoc commented 2 years ago

I couldn't run because include files are missing. By quickly looking the data file: SWAT, SWFN, THCCOIl, THCGAS and THCWATER keywords are not needed. Note from the manual that THCONR is the reservoir rock plus fluid thermal conductivity, and is not supposed to be used in combination with other THC words. Also in VISCREF, item 2 and 3 not needed, since the model does not contains gas dissolved as activated by the DISGAS . You could simply use:

VISCREF
-- --   REF      REF 
-- -- PRESSURE   GOR 
         426.2     0.0 / 
OPMUSER commented 2 years ago

@goncalvesmachadoc

Tried your suggestions and still got my favorite error message.

Report step  0/13 at day 0/366, date = 01-Jan-2020
flow: /home/david/OPM/GitHub/opm-material/opm/material/fluidsystems/BlackOilFluidSystem.hpp:1532: static short int Opm::BlackOilFluidSystem<Scalar, IndexTraits>::canonicalToActivePhaseIdx(unsigned int) [with Scalar = double; IndexTraits = Opm::BlackOilDefaultIndexTraits]: Assertion `phaseIsActive(phaseIdx)' failed.
[david-VirtualBox:02321] *** Process received signal ***
[david-VirtualBox:02321] Signal: Aborted (6)
[david-VirtualBox:02321] Signal code:  (-6)
[david-VirtualBox:02321] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7fed07879520]
[david-VirtualBox:02321] [ 1] /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c)[0x7fed078cda7c]
[david-VirtualBox:02321] [ 2] /lib/x86_64-linux-gnu/libc.so.6(raise+0x16)[0x7fed07879476]
[david-VirtualBox:02321] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3)[0x7fed0785f7f3]
[david-VirtualBox:02321] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x2871b)[0x7fed0785f71b]
[david-VirtualBox:02321] [ 5] /lib/x86_64-linux-gnu/libc.so.6(+0x39e96)[0x7fed07870e96]
[david-VirtualBox:02321] [ 6] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x15e0211)[0x55e2a412d211]
[david-VirtualBox:02321] [ 7] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x165808b)[0x55e2a41a508b]
[david-VirtualBox:02321] [ 8] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x166bfb4)[0x55e2a41b8fb4]
[david-VirtualBox:02321] [ 9] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x166c3c6)[0x55e2a41b93c6]
[david-VirtualBox:02321] [10] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x169515e)[0x55e2a41e215e]
[david-VirtualBox:02321] [11] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x1695b4b)[0x55e2a41e2b4b]
[david-VirtualBox:02321] [12] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x16861de)[0x55e2a41d31de]
[david-VirtualBox:02321] [13] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x15f60a1)[0x55e2a41430a1]
[david-VirtualBox:02321] [14] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x235cb5)[0x55e2a2d82cb5]
[david-VirtualBox:02321] [15] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fed07860d90]
[david-VirtualBox:02321] [16] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fed07860e40]
[david-VirtualBox:02321] [17] /home/david/OPM/GitHub/opm-simulators/build/bin/flow(+0x34a7d5)[0x55e2a2e977d5]
[david-VirtualBox:02321] *** End of error message ***

Test deck attached with include files (hopefully). TEMPI4_RTEMPVD_FLOW4-DBx.zip

EdmundStephens commented 2 years ago

Hi, grateful for the debate, I did not know THCONR nominally included the THCOIL/WATER/GAS ... good to know thank you.

I added the oil and gas phases because Flow does not accept a single phase WATER model (and GAS/DISGAS only becasue I cut+pasted from another model with a PVTO table).

This is a separate issue really and I will post this separately so it is more trackable, the attached models are WATER only. Runs in Eclipse but gives the following error in Flow.

No suitable configuration found, valid are Twophase, polymer, foam, brine, solvent, energy, and blackoil.

WATER1_FLOW.zip WATER1_ECL.zip

EdmundStephens commented 2 years ago

Also further to this debate, I made a version with only WATER+OIL (PVDO table) and replaced the Shallow.grdecl grid with a simple cartesian box SimBox.grdecl.

The result still shows the same convergence error. Hope this helps focus your bug hunting!

TEMPI_SIMBOX.zip

OPMUSER commented 2 years ago

For the record TEMPI_SIMBOX has the same issue running under 2022-04, solver convergence. WATER_FLOW just crashes

===============Saturation Functions Diagnostics===============

System:  Single phase Water system. Nothing to check

================ Starting main simulation loop ===============

Report step  0/13 at day 0/366, date = 01-Jan-2020
OPM Flow Process 14892 Segmentation Fault - Job Aborted

This is just a water case no TTEMP/THERMAL option.

OPMUSER commented 2 years ago

@EdmundStephens

image

Couple points to improve the general modeling experience, not related to the OPM Flow bugs:

  1. We don't model the sea or ocean in the simulation grid, as this very large body of water is not influenced by the wellbore fluids. Instead we model the impact of the wellbore fluids by the seawater, as the the seawater has a cooling effect on the produced fluids, especially in deep water fields. So you can eliminate seawater part of the model. Then you need to build vertical lift tables that take this into account using the appropriate software, and then incorporate the VFP curves via the VFPPROD keyword. The industry standard for this Nodal Analysis software is Petroleum Experts' PROSPER program. Having said all this, OPM Flow does not support the WHTEMP keyword needed allocate the temperature VFPPROD table to the well, and therefore the numerics as well. But you can still try this approach with E100.
  2. Secondly the grid is an issue even if we eliminate the sea water blocks, because of the non-orthogonality of the cells. I assume the shape of the model is being constrained by faults (or other geological discontinuities), if that is the case you need to use the zig-zag fault option when building the model in the earth modelling software. The figure below shows a non-orthogonal grid were the actual fault traces are being honored, notice how distorted the grid is in this instance. If use the zig-zag fault option instead the grid is orthogonal (more or less). This reduces grid orientation effects as well poor numerical performance.

image

Hope this is helpful.

OPMUSER commented 2 years ago

For the record TEMPI_SIMBOX has the same issue running under 2022-10, solver convergence. WATER_FLOW just crashes

I mean 2022-04 in the above, but same problems occur with master as well.

EdmundStephens commented 2 years ago

RE the grid, you are right the sea water layers are unnecessary as well as a nuisance. The idea we have is to represent both the hydraulic source as well as thermal heat sink, i.e. effective constant pressure. This is better modelled as an aquifer attached to the surface outcrop, and suitably inflating the pore volumes of the top most blocks sufficient to effectively fix these to 40 F.

The line of enquiry was prompted in part from the geomechanical modelling in the CCUS work. Those models (in Visage) require the underburden as well as 'sideburdens' to be modelling as those contribute to the poroelastic response of the formation. I was surprised to observe the geomechanical model had been extended to 80 km depth of underburden. Likewise in the geothermal context, there is some uplift form a bottom heat flux in equilibrium with a surface constant pressure ... and for the shallow geothermal concept (e.g. 100 - 1000m) with low temperatures, this is proportionally much more important than the deep high temperate plays at 3 - 5 km or deeper.