Lorenz reader improvements.

[NAThompson]

Ok friends, got a quick and hopefully easy problem with this one:

Whenever I run this, I hit an exception:

➜  build git:(lorenz_ode) ✗ ./bin/lorenz_reader 
Simulation performed with σ = 10, β = 2.66667, and ρ = 28.
The data should be interpreted as a 7D array of structs {tᵢ, xᵢ, yᵢ, zᵢ, ẋᵢ, ẏᵢ, żᵢ}.
Solutions were computed with an absolute error goal of 1e-05
Caught exception reading Lorenz ODE solution: ERROR: offset 0 from steps start 1 in variable u001 is beyond the largest available step = 1, check Variable SetStepSelection argument stepsCount (random access), or number of BeginStep calls (streaming), in call to Get

If I add a BeginStep()/EndStep() i.e.,

double* d = v[0].data();
adios_engine.BeginStep();
adios_engine.Get(key, d, adios2::Mode::Sync);
adios_engine.EndStep();

it gets even weirder:

➜  build git:(lorenz_ode) ✗ ./bin/lorenz_reader 
Simulation performed with σ = 10, β = 2.66667, and ρ = 28.
The data should be interpreted as a 7D array of structs {tᵢ, xᵢ, yᵢ, zᵢ, ẋᵢ, ẏᵢ, żᵢ}.
Solutions were computed with an absolute error goal of 1e-05
We do not have a state variable u001
We do not have a state variable u010
We do not have a state variable u011
We do not have a state variable u100
We do not have a state variable u101
We do not have a state variable u110
We do not have a state variable u111

[williamfgc]

Try using bpls and see if the file is correct

[NAThompson]

Looks ok:

$ bpls lorenz.bp -lav
File info:
  of variables:  8
  of attributes: 5
  statistics:    Min / Max

  string   interpretation  attr   = "7D array of structs {tᵢ, xᵢ, yᵢ, zᵢ, ẋᵢ, ẏᵢ, żᵢ}"
  double   u000            {1799} = 0 / 10.0221
  double   u001            {1799} = -2.66667 / 10.0221
  double   u010            {130431} = -388.691 / 409.646
  double   u011            {145936} = -376.305 / 397.53
  double   u100            {131369} = -388.446 / 409.421
  double   u101            {146608} = -375.409 / 396.659
  double   u110            {131950} = -388.799 / 409.766
  double   u111            {149506} = -373.8 / 395.091
  double   β              attr   = 2.66667
  double   ρ              attr   = 28
  double   σ              attr   = 10
  double   ‖û-u‖      attr   = 1e-05

I reran both the writer and reader under -fsanitize=address -fsanitize=undefined; didn't get anything. Also recompiled ADIOS2 on top of master with -fsanitize=address -fsanitize=undefined; nothing.

[williamfgc]

try bpls lorenz.bp -lavd to see if it can access the data

[NAThompson]

Nope, it can't. Same exception I found in the reader:

File info:
  of variables:  8
  of attributes: 5
  statistics:    Min / Max 

  string   interpretation  attr   = "7D array of structs {tᵢ, xᵢ, yᵢ, zᵢ, ẋᵢ, ẏᵢ, żᵢ}"
  double   u000            {1799} = 0 / 10.0221
    (   0)    0 0 0 0 0 0
    (   6)    0 0.0391487 0 0 0 0
   ....
  double   u001            {1799} = -2.66667 / 10.0221

bpls caught an exception
ERROR: offset 0 from steps start 1 in variable u001 is beyond the largest available step = 1, check Variable SetStepSelection argument stepsCount (random access), or number of BeginStep calls (streaming), in call to Get

[pnorbert]

Arggh, I can see, you do a new step for each variable. u000 in step 1 u001 in step 2, and so on

The BP reader somehow does not like this. It should...

[pnorbert]

I meant, you do a new step at writing.

[NAThompson]

@pnorbert : I just validated that your fix works! Thanks!

I guess this should work, but in some sense I've abused the abstraction of a Step, and hence I should remove it from the example.

[pnorbert]

What fix?

[pnorbert]

I can read the data if I modify the reader to go step by step. It finds one variable at a time.

void read_solution()
{
    using Real = double;
    adios2::ADIOS adios;
    adios2::IO io = adios.DeclareIO("myio");
    adios2::Engine adios_engine = io.Open("lorenz.bp", adios2::Mode::Read);
    bool firstStep = true;

    while (true)
    {
        adios2::StepStatus read_status = adios_engine.BeginStep(adios2::StepMode::Read);
        if (read_status != adios2::StepStatus::OK)
        {
            break;
        }

        int stepSimOut = adios_engine.CurrentStep();

        if (firstStep)
        {
            auto sigma_att = io.InquireAttribute<Real>("σ");
            std::cout << "Simulation performed with ";
            if (sigma_att)
            {
                Real sigma = sigma_att.Data()[0];
                std::cout << "σ = " << sigma << ", ";
            }

            auto beta_att = io.InquireAttribute<Real>("β");
            if (beta_att)
            {
                Real beta = beta_att.Data()[0];
                std::cout << "β = " << beta << ", ";
            }

            auto rho_att = io.InquireAttribute<Real>("ρ");
            if (rho_att)
            {
                Real rho = rho_att.Data()[0];
                std::cout << "and ρ = " << rho << ".\n";
            }

            auto interpretation_att = io.InquireAttribute<std::string>("interpretation");
            if (interpretation_att)
            {
                std::string interpretation = interpretation_att.Data()[0];
                std::cout << "The data should be interpreted as a " << interpretation << ".\n";
            }

            auto absolute_error_att = io.InquireAttribute<Real>("‖û-u‖");
            if (absolute_error_att)
            {
                std::cout << "Solutions were computed with an absolute error goal of "
                          << absolute_error_att.Data()[0] << "\n";
            }

            // Check that std::vector<std::array<Real, 7>> has the proper memory layout.
            static_assert(
                sizeof(std::array<Real, 7>) == 7 * sizeof(Real),
                "The std::array on your system does not have the proper layout to be correctly "
                "deserialized in ADIOS2.");
        }

        std::cout << "=====================================\n"
                  << "Step: " << stepSimOut << "\n";

        std::vector<std::array<Real, 7>> v;
        auto const &m = io.AvailableVariables();
        for (auto const &[key, val] : io.AvailableVariables())
        {
            std::cout << "--------------------------------\n"
                      << "Name: " << key << "\n";

            for (const auto &parameter : val)
            {
                std::cout << "\t" << parameter.first << ": " << parameter.second << "\n";
            }
            auto state_variable = io.InquireVariable<Real>(key);
            if (!state_variable)
            {
                std::cerr << "We do not have a state variable " << key << "\n";
                continue;
            }
            const auto shape = state_variable.Shape();
            if (shape.size() != 1)
            {
                std::cerr << "Expected 1D layout.\n";
                continue;
            }
            if ((shape[0] % 7) != 0)
            {
                std::cerr << "The size of a 7D array of structs must be a multiple of 7.\n";
                continue;
            }
            size_t states = shape[0] / 7;
            v.resize(states);

            double *d = v[0].data();
            adios_engine.Get(key, d, adios2::Mode::Sync);
        }
        firstStep = false;
    }
    adios_engine.Close();
}

[NAThompson]

What fix?

Well, it works if I remove BeginStep()/EndStep() from the writer.

[pnorbert]

Ah, single step in writer works too, but you have a single output at the end of the writer. All your results are kept in memory until the end. Depening on what you want to do with this example that is either fine or not. E.g. streaming out solutions as you find them and process them in a reader will not work this way

[NAThompson]

I explicitly put the writer in Sync mode; will this not obviate that problem?

[williamfgc]

I think you're both right, @pnorbert means the copy to the adios2 buffer, @NAThompson means the user memory is released with sync. Maybe this illustrates how we can go from single step to multiple steps depending on what the users needs.

[pnorbert]

It is a very big problem if users think Sync will make file IO, and it will make steps in the output. Nick, Sync only makes sure that you can reuse the pointer after Put(), but it does promise nothing about writing to disk. In practice, it is just copied to an output buffer.

[pnorbert]

Question 1: how are you going to parallelize this example?

• parallelize finding one solution and write that incrementally?
• parallelize over finding different solutions?

Question 2: do you want to stream out the solutions as you find them, or do you want to just store them as most convenient?

[williamfgc]

It is a very big problem if users think Sync will make file IO, and it will make steps in the output.

@pnorbert I agree, since this is for tutorials I think we should present adios no different from other buffered I/O libraries (fstream, stdio) as write/fwrite or << don't promise anything until flush (adios2::Engine::Flush) or std::endl (default adios2::Engine::EndStep).

[NAThompson]

parallelize over finding different solutions?

Definitely want to parallelize over finding different solutions.

do you want to stream out the solutions as you find them, or do you want to just store them as most convenient?

My goal right now is to just write it to a .bp file. My eventual goal is the inline engine + rendering in Paraview.

[NAThompson]

since this is for tutorials I think we should present adios no different from other buffered I/O libraries (fstream, stdio) as write/fwrite or << don't promise anything until flush (adios2::Engine::Flush) or std::endl (default adios2::Engine::EndStep).

Would it make sense to make an explicit call to adios2::Engine::Flush in this example?

[williamfgc]

Would it make sense to make an explicit call to adios2::Engine::Flush in this example?

Only if that's what is intended.

[pnorbert]

No, it wouldn't make sense, more like make confusion. Adios Flush() does not do steps, just empties buffers to disk. Data would not be readable until the file is Close()-d and all data and metadata is written to disk (the step is completed).

[pnorbert]

In the inline engine, do you want to process each solution separately, or make one visualization on the entire solution set?

[NAThompson]

In the inline engine, do you want to process each solution separately, or make one visualization on the entire solution set?

For now, I want to process each solution separately. But it's closer in spirit to the particle advection to make a single visualization for the entire solution set, so I'd like to move in that direction.

This is a good representation of what I'd like to do.

[NAThompson]

No, it wouldn't make sense, more like make confusion. Adios Flush() does not do steps, just empties buffers to disk. Data would not be readable until the file is Close()-d and all data and metadata is written to disk (the step is completed).

I added a Flush() to the writer:

const std::string variable = "u" + std::to_string(i) + std::to_string(j) + std::to_string(k);
 auto state_variable = io.DefineVariable<Real>(variable, {7*skeleton.size()}, {0}, {7*skeleton.size()}, adios2::ConstantDims);
adios_engine.Put(state_variable, p, adios2::Mode::Sync);
adios_engine.Flush();

and then none of my attributes showed up:

build git:(lorenz_ode) ✗ bpls lorenz.bp -lav  
File info:
  of variables:  1
  of attributes: 0
  statistics:    Min / Max 

  double   u111  {149506} = -373.8 / 395.091

So I guess I'm doing something truly bizarre here and hence made the Flush() call moot point.

[williamfgc]

@NAThompson I'd check with @pnorbert about the overall goal of the miniapp. adios is step based, attributes are not registered until the first EndStep. Flush is a special use case (even in standard buffered I/O is discouraged, think std::endl being expensive). Your I/O should be framed in steps, rather than variables as individual units of data movement.

[NAThompson]

adios is step based, attributes are not registered until the first EndStep.

That makes sense, but strangely I get the attributes registered without calls to BeginStep()/EndStep(); only Flush() causes the attributes to not register. (Not sure how interesting this is since Flush() is not encouraged.

[williamfgc]

They are registered at EndStep or Close (which call EndStep if not called). It's not just Flush, it's Flush + Put in sync mode the result of what you're seeing. I'm not familiar with the inline engine, but why not keeping everything under BeginStep/EndStep on both writer and reader and remove Flush? The idea of BP4 is to do streaming through files, so the code would be portable across engines. Perhaps, I'm missing something in your spec.

[pnorbert]

I think you need a single BeginStep/Endstep pair in which you write all solutions. EndStep is required for the inline engine to work. Having one step will let you process all solutions in a single step in your visualization tool.

[pnorbert]

But if you can do some processing for every solution separately, and than do something at the end, that would be nicer for a tutorial example (many steps). You would need to signal a "last" step to the inline engine. You need to ask Kitware though if the inline engine steps are in one memory space or if the reader starts afresh at every step.

[NAThompson]

So does this example, in its current state, use ADIOS in an idiomatic way?

[NAThompson]

@williamfgc : Do you mind merging this? It fixes quite a few problems with the already-merged code.