MPI distributed renderer's renderFrame hangs when more than one instance is ran on the same node.

[acdemiralp]

• Everything is ok with -n 1 -hosts localhost and -ppn 1 -hosts .... Hangs with -n 2 -hosts localhost.
• In the test case, MPI is already initialized before the first call to Ospray.
• Tracking it further. I'll update.

[jeffamstutz]

We are keen to know what you find, as multiple ranks on a single node should work (though not optimal for rendering performance).

3 quick questions for my own clarity:

1. What hardware is this running on (Xeon/KNL/laptop)?
2. What device are you using (mpi_offload or mpi_distributed)?
3. Is this a custom app, a VTK app (i.e. ParaView or VisIt), or one of our viewers?

The answer to (2) can have very different issues associated with them, as they have very different usages.

Even if you end up figuring it out on your own, it would be helpful to know a run time environment that resulted in a problem. :)

Cheers, Jeff

[acdemiralp]

Dear Jeff,

1- Laptop, Intel Core i5-8350U CPU@1.70GHz 2- The mpi_distributed device. I partition the problem in 3 dimensions, and each rank adds its own block of geometry (with appropriate offsets according to rank) to a model. I assume things are just as in non-mpi mode except this, and call renderFrame on all ranks, however only store the output image of rank 0. 3- It is a custom app that does particle tracing and then renders the streamlines in-situ with Ospray.

I could trace that it does not get past the condition wait in DistributedFrameBuffer.cpp:345:

auto startWaitFrame = high_resolution_clock::now();
std::unique_lock<std::mutex> lock(mutex);
frameDoneCond.wait(lock, [&]{
  return frameIsDone;
});

Any ideas? The Ospray part of my code is only about 150 lines. Do you see something stupidly off below? :

#include <prs/stages/raytracer.hpp>

namespace prs
{
raytracer::raytracer (const std::size_t thread_count, boost::mpi::communicator* communicator) : communicator_(communicator)
{
  if (ospLoadModule("ispc") != OSP_NO_ERROR)
    throw std::runtime_error("Failed to load OSPRay ISPC module");
  if (ospLoadModule("mpi") != OSP_NO_ERROR)
    throw std::runtime_error("Failed to load OSPRay MPI module");

  const auto device = ospray::cpp::Device("mpi_distributed");
  ospDeviceSetStatusFunc(device.handle(), [ ] (const char* message) { std::cout << message << "\n"; });
  device.set       ("numThreads" , thread_count);
  device.set       ("setAffinity", 1           );
  device.set       ("masterRank" , 0           );
#if _DEBUG
  device.set       ("logLevel"   , 2           );
  device.set       ("logOutput"  , "cout"      );
  device.set       ("errorOutput", "cerr"      );
#endif
  device.commit    ();
  device.setCurrent();

  auto material = ospray::cpp::Material("raycast", "OBJMaterial");
  material.set   ("Ks", 0.7F, 0.7F, 0.7F);
  material.set   ("Ns", 10.0F);
  material.commit();

  geometry_   .emplace_back(std::make_unique<ospray::cpp::Geometry>("streamlines"));
  vertex_data_.emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_FLOAT3A, nullptr)); vertex_data_.back()->commit();
  color_data_ .emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_FLOAT4 , nullptr)); color_data_ .back()->commit();
  index_data_ .emplace_back(std::make_unique<ospray::cpp::Data>(0, OSP_INT    , nullptr)); index_data_ .back()->commit();

  geometry_[0]->set        ("vertex"      , *vertex_data_.back());
  geometry_[0]->set        ("vertex.color", *color_data_ .back());
  geometry_[0]->set        ("index"       , *index_data_ .back());
  geometry_[0]->set        ("radius"      , 0.1F);
  geometry_[0]->setMaterial(material);
  geometry_[0]->commit     ();

  model_ = std::make_unique<ospray::cpp::Model>();
  model_->addGeometry(*geometry_[0]);
  model_->set        ("id", communicator_->rank());
  model_->commit     ();

  camera_ = std::make_unique<ospray::cpp::Camera>("perspective");
  camera_->set   ("fovy"      , 68.0F);
  camera_->set   ("imageStart", ospcommon::vec2f{0.0F, 1.0F});
  camera_->set   ("imageEnd"  , ospcommon::vec2f{1.0F, 0.0F});
  camera_->commit();

  auto ambient_light = ospray::cpp::Light("ambient");
  ambient_light.set   ("intensity", 0.2F);
  ambient_light.commit();
  const auto ambient_handle = ambient_light.handle();

  auto distant_light = ospray::cpp::Light("distant");
  distant_light.set   ("direction"      , 1.0F, 1.0F, -0.5F);
  distant_light.set   ("color"          , 1.0F, 1.0F,  0.8F);
  distant_light.set   ("intensity"      , 0.8F);
  distant_light.set   ("angularDiameter", 1.0F);
  distant_light.commit();
  const auto distant_handle = distant_light.handle();

  std::vector<OSPLight> lights_list = {ambient_handle, distant_handle};
  lights_ = std::make_unique<ospray::cpp::Data>(lights_list.size(), OSP_LIGHT, lights_list.data());
  lights_->commit();

  renderer_ = std::make_unique<ospray::cpp::Renderer>("mpi_raycast");
  renderer_->set   ("aoSamples"            , 1       );
  renderer_->set   ("aoTransparencyEnabled", false   );
  renderer_->set   ("oneSidedLighting"     , true    );
  renderer_->set   ("spp"                  , 1       );
  renderer_->set   ("bgColor"              , 1.0F, 1.0F, 1.0F, 1.0F);
  renderer_->set   ("camera"               , *camera_);
  renderer_->set   ("lights"               , *lights_);
  renderer_->set   ("model"                , *model_ );
  renderer_->commit();

  framebuffer_ = std::make_unique<ospray::cpp::FrameBuffer>(ospcommon::vec2i(32, 32), OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FB_ACCUM);
  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}
raytracer::~raytracer()
{  
  framebuffer_.reset();
  lights_     .reset();
  index_data_ .clear();
  color_data_ .clear();
  vertex_data_.clear();
  geometry_   .clear();
  model_      .reset();
  camera_     .reset();
  renderer_   .reset();
  ospShutdown();
}

void  raytracer::set_streamlines(streamlines<>*                      streamlines)
{
  auto material = ospray::cpp::Material("scivis", "OBJMaterial");
  material.set   ("Ks", 0.7F, 0.7F, 0.7F);
  material.set   ("Ns", 10.0F);
  material.commit();

  model_ = std::make_unique<ospray::cpp::Model>();
  geometry_   .clear();
  vertex_data_.clear();
  color_data_ .clear();
  index_data_ .clear();
  geometry_   .reserve(streamlines->chunks.size());
  vertex_data_.reserve(streamlines->chunks.size());
  color_data_ .reserve(streamlines->chunks.size());
  index_data_ .reserve(streamlines->chunks.size());

  if(communicator_->rank() == 0)
  {
  for(auto& chunk : streamlines->chunks)
  {
    if (chunk.indices.empty())
      continue;

    const auto vertex   = vertex_data_.emplace_back(new ospray::cpp::Data(chunk.vertices.size(), OSP_FLOAT3A, chunk.vertices.data(), OSP_DATA_SHARED_BUFFER)).get(); vertex->commit();
    const auto color    = color_data_ .emplace_back(new ospray::cpp::Data(chunk.colors  .size(), OSP_FLOAT4 , chunk.colors  .data(), OSP_DATA_SHARED_BUFFER)).get(); color ->commit();
    const auto index    = index_data_ .emplace_back(new ospray::cpp::Data(chunk.indices .size(), OSP_INT    , chunk.indices .data(), OSP_DATA_SHARED_BUFFER)).get(); index ->commit();
    const auto geometry = geometry_   .emplace_back(new ospray::cpp::Geometry("streamlines")).get();
    geometry->setMaterial(material);
    geometry->set        ("vertex"      , *vertex);
    geometry->set        ("vertex.color", *color );
    geometry->set        ("index"       , *index );
    geometry->set        ("radius"      , streamlines->radius);
    geometry->commit     ();
    model_  ->addGeometry(*geometry);
  }
  model_->set   ("id", communicator_->rank());
  model_->commit();
  }

  renderer_->set   ("model", *model_);
  renderer_->commit();

  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}
void  raytracer::set_camera     (const camera&                       camera     )
{
  camera_->set   ("pos", ospcommon::vec3f{camera.position[0], camera.position[1], camera.position[2]});
  camera_->set   ("dir", ospcommon::vec3f{camera.forward [0], camera.forward [1], camera.forward [2]});
  camera_->set   ("up" , ospcommon::vec3f{camera.up      [0], camera.up      [1], camera.up      [2]});
  camera_->commit();

  framebuffer_->clear (OSP_FB_COLOR | OSP_FB_ACCUM);
}
void  raytracer::set_image_size (const std::array<std::uint64_t, 2>& image_size )
{
  framebuffer_size_ = image_size;

  camera_->set   ("aspect", framebuffer_size_[0] / static_cast<float>(framebuffer_size_[1]));
  camera_->commit();

  framebuffer_ = std::make_unique<ospray::cpp::FrameBuffer>(
    ospcommon::vec2i(static_cast<std::int32_t>(framebuffer_size_[0]), static_cast<std::int32_t>(framebuffer_size_[1])),
    OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FB_ACCUM);
  framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM);
}

void  raytracer::trace          (const std::size_t                   iterations )
{
  std::cout << "I will get stuck soon." << std::endl;
  for (std::size_t i = 0; i < iterations; ++i)
    renderer_->renderFrame(*framebuffer_, OSP_FB_COLOR | OSP_FB_ACCUM);
  std::cout << "No?" << std::endl;
}
image raytracer::serialize      ()
{
  image image;
  const auto bytes = static_cast<std::uint8_t*>(framebuffer_->map(OSP_FB_COLOR));
  image.set_data    (bytes, framebuffer_size_[0] * framebuffer_size_[1] * sizeof(std::array<std::uint8_t, 4>));
  image.mutable_size()->Resize(2, 0);
  image.mutable_size()->Set   (0, framebuffer_size_[0]);
  image.mutable_size()->Set   (1, framebuffer_size_[1]);
  framebuffer_->unmap(bytes);
  return image;
}
}

[Twinklebear]

Hey @acdemiralp ,

From looking through the code I think the issue is your model and geometry setup in set_streamlines. The problem with the code below is that only rank 0 specifies the streamline geometry, and the other ranks have empty models with no data. When running with the distributed device each rank needs to specify its own data independently, we don't broadcast out data or API calls like in the offload device. So instead each rank should run the loop to setup the geometry in that function, but will specify different geometry from its local chunk. A possible implementation would be as follows, where now chunk actually refers to a distinct streamline piece which is computed by and owned by the rank. You may also want to look at how the scene is setup in the MPIDistributed Spheres Tutorial.

// chunk is a unique streamlines<> owned by this rank only
if (!chunk.indices.empty()) {
    const auto vertex   = vertex_data_.emplace_back(new ospray::cpp::Data(chunk.vertices.size(), OSP_FLOAT3A, chunk.vertices.data(), OSP_DATA_SHARED_BUFFER)).get(); vertex->commit();
    const auto color    = color_data_ .emplace_back(new ospray::cpp::Data(chunk.colors  .size(), OSP_FLOAT4 , chunk.colors  .data(), OSP_DATA_SHARED_BUFFER)).get(); color ->commit();
    const auto index    = index_data_ .emplace_back(new ospray::cpp::Data(chunk.indices .size(), OSP_INT    , chunk.indices .data(), OSP_DATA_SHARED_BUFFER)).get(); index ->commit();
    const auto geometry = geometry_   .emplace_back(new ospray::cpp::Geometry("streamlines")).get();
    geometry->setMaterial(material);
    geometry->set        ("vertex"      , *vertex);
    geometry->set        ("vertex.color", *color );
    geometry->set        ("index"       , *index );
    geometry->set        ("radius"      , streamlines->radius);
    geometry->commit     ();
    model_  ->addGeometry(*geometry);
}
model_->set   ("id", communicator_->rank());
// If the model bounds are not disjoint, also specify region.lower 
// and region.upper here
model_->commit();
renderer_->set   ("model", *model_);
renderer_->commit();

I think that's likely the reason it hangs, but some other possible issues may also be worth looking at if that doesn't fix it.

Another possible issue is that there might be some overlap between the bounding boxes of the geometry specified on each rank, if you have some ghost zones or something. It might be useful to check the list of bounding boxes and model information that the renderer is getting. If you increase the log level to 3 you should see a list of bounding boxes and list of ranks that own them (printed here). If there's any overlap this could be an issue, or if some ranks bounds are empty (a box3f(inf, -inf)) the compositor may not figure out what pixels they touch properly. To fix this you can override the model bounds on each rank by setting the region.lower and region.upper vec3f parameters on the models, to clip the bounds to just the grid cell they own.

Another possible issue is if you're creating the framebuffer on one rank but not the others, or are out of order in how you call it. If the ranks get out of sync here they won't be communicating with the right framebuffer object on the other ranks, so making the 32x32 pixel framebuffer in the constructor may be an issue.

Another thing that may not be a bug but is probably good to remove is that in the constructor you setup a scene and geometry with empty vertex values/etc, I'm not sure if that may cause some issue as well with figuring out the object's bounds locally. Same with making the 32x32 empty framebuffer.

Since you're using AO on distributed data you might also see some artifacts if you don't replicate the data around to build ghost geometry as well for computing the AO/shadows.

[acdemiralp]

Dear Will,

Thank you very much for the detailed response.

I apologize, the first error you noticed was for debugging purposes after I started getting this problem. It exists even when if (rank == 0) is removed.

I am looking into the other issues you mentioned and going to update soon.

[acdemiralp]

Getting there. Based on Will's suggestions, I have placed some communicator barriers around geometry construction and renderFrame. Now rank 0 yields me the following image: The expected (single process) image:

• The framebuffer compositor seems to be confused?
• But all tiles that did make it are correct, they perfectly overlay with the expected image.
• FYI the way the data is distributed is by a single cut in the middle of the Y dimension.

[Twinklebear]

Interesting, not hanging is good! It sounds like the framebuffer handles are sync'd up properly now, so the renderer can at least communicate properly between processes.

My guess as to why you're seeing roughly half of the data missing is due to only rank 0 setting up the geometry in your set_streamlines function. Since only rank 0 has any data or volume data the others will just render some background. Though in this case it looks like a volume dataset, which I don't see setup in your code? Could you share that setup as well? For the volume I would also recommend checking out the MPI Distributed Structured Volume Tutorial which will show how to setup a distributed volume dataset, offset the bricks and clip ghost voxels by overriding the model's region bounds. The background color differences may come from setting different bgColor on each process, or not setting it on some of them.

[acdemiralp]

I decided to try out Ospray 1.7.3 just as a last hour effort at work, without any changes to code and it worked, showing me the image I expected to see:

Due to time constraints (EGPGV), I cannot spend more time on fixing what's wrong with the latest distributed renderer at the moment, since I still have to make that mid-line disappear and write a half-paper on top of that in 11 days.

I will get back onto it after the deadline, because your library is awesome in many ways and I wanna make it better, although it really makes me sad when behavior changes randomly version to version, resulting in 2 research paper ragequits until today.

@Twinklebear :

• The rank==0 was due to debugging as I mentioned earlier. It nevertheless gave the earlier results you mentioned without that if statement.
• Those are not volumes but very tiny streamlines seen from afar.
• This code works in 1.7.3 but not in 1.8.0:

  
  #include <prs/stages/raytracer.hpp>

namespace prs { raytracer::raytracer (boost::mpi::communicator communicator, const std::size_t threadcount) : communicator(communicator), rawcommunicator(communicator) { if (ospLoadModule("ispc") != OSP_NO_ERROR) throw std::runtime_error("Failed to load OSPRay ISPC module"); if (ospLoadModule("mpi") != OSP_NO_ERROR) throw std::runtime_error("Failed to load OSPRay MPI module");

const auto device = ospray::cpp::Device("mpi_distributed"); ospDeviceSetStatusFunc(device.handle(), [ ] ( const char message) { std::cout << message << "\n"; }); ospDeviceSetErrorFunc (device.handle(), [ ] (OSPError error, const char message) { std::cout << message << "\n"; }); device.set ("numThreads" , thread_count ); device.set ("setAffinity" , 1 ); device.set ("masterRank" , 0 ); device.set ("worldCommunicator", &rawcommunicator); //#if _DEBUG device.set ("logLevel" , 3 ); device.set ("logOutput" , "cout" ); device.set ("errorOutput" , "cerr" ); //#endif device.commit (); device.setCurrent();

auto material = ospray::cpp::Material("scivis", "OBJMaterial"); material.set ("Ks", 0.7F, 0.7F, 0.7F); material.set ("Ns", 10.0F); material.commit();

geometry_ .emplace_back(std::make_unique("streamlines")); vertexdata.emplace_back(std::make_unique(0, OSP_FLOAT3A, nullptr)); vertexdata.back()->commit(); colordata .emplace_back(std::make_unique(0, OSP_FLOAT4 , nullptr)); colordata .back()->commit(); indexdata .emplace_back(std::make_unique(0, OSP_INT , nullptr)); indexdata .back()->commit();

geometry_[0]->set ("vertex" , vertexdata.back()); geometry_[0]->set ("vertex.color", colordata .back()); geometry_[0]->set ("index" , *indexdata .back()); geometry[0]->set ("radius" , 0.1F); geometry[0]->setMaterial(material); geometry_[0]->commit ();

model_ = std::makeunique(); model->addGeometry(*geometry[0]); model->set ("id", communicator->rank()); model->commit ();

camera_ = std::makeunique("perspective"); camera->set ("fovy" , 68.0F); camera->set ("imageStart", ospcommon::vec2f{0.0F, 1.0F}); camera->set ("imageEnd" , ospcommon::vec2f{1.0F, 0.0F}); camera_->commit();

auto ambient_light = ospray::cpp::Light("ambient"); ambient_light.set ("intensity", 0.2F); ambient_light.commit(); const auto ambient_handle = ambient_light.handle();

auto distant_light = ospray::cpp::Light("distant"); distant_light.set ("direction" , 1.0F, 1.0F, -0.5F); distant_light.set ("color" , 1.0F, 1.0F, 0.8F); distant_light.set ("intensity" , 0.8F); distant_light.set ("angularDiameter", 1.0F); distant_light.commit(); const auto distant_handle = distant_light.handle();

std::vector lights_list = {ambient_handle, distanthandle}; lights = std::make_unique(lights_list.size(), OSP_LIGHT, lightslist.data()); lights->commit();

renderer_ = std::make_unique("mpiraycast"); renderer->set ("aoSamples" , 1 ); renderer->set ("aoTransparencyEnabled", false ); renderer->set ("oneSidedLighting" , true ); renderer->set ("spp" , 1 ); renderer->set ("bgColor" , 1.0F, 1.0F, 1.0F, 1.0F); renderer->set ("camera" , *camera); renderer->set ("lights" , *lights); renderer->set ("model" , *model ); renderer_->commit();

framebuffer_ = std::make_unique(ospcommon::vec2i(32, 32), OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FBACCUM); framebuffer->clear(OSP_FB_COLOR | OSP_FBACCUM); } raytracer::~raytracer() {
framebuffer.reset(); lights_ .reset(); indexdata .clear(); colordata .clear(); vertexdata.clear(); geometry .clear(); model .reset(); camera .reset(); renderer .reset(); ospShutdown(); }

void raytracer::set_streamlines(streamlines<>* streamlines) { auto material = ospray::cpp::Material("scivis", "OBJMaterial"); material.set ("Ks", 0.7F, 0.7F, 0.7F); material.set ("Ns", 10.0F); material.commit();

model_ = std::makeunique(); geometry .clear(); vertexdata.clear(); colordata .clear(); indexdata .clear(); geometry_ .reserve(streamlines->chunks.size()); vertexdata.reserve(streamlines->chunks.size()); colordata .reserve(streamlines->chunks.size()); indexdata .reserve(streamlines->chunks.size());

for(auto& chunk : streamlines->chunks) { if (chunk.indices.empty()) continue;

const auto vertex   = vertex_data_.emplace_back(new ospray::cpp::Data(chunk.vertices.size(), OSP_FLOAT3A, chunk.vertices.data(), OSP_DATA_SHARED_BUFFER)).get(); vertex->commit();
const auto color    = color_data_ .emplace_back(new ospray::cpp::Data(chunk.colors  .size(), OSP_FLOAT4 , chunk.colors  .data(), OSP_DATA_SHARED_BUFFER)).get(); color ->commit();
const auto index    = index_data_ .emplace_back(new ospray::cpp::Data(chunk.indices .size(), OSP_INT    , chunk.indices .data(), OSP_DATA_SHARED_BUFFER)).get(); index ->commit();
const auto geometry = geometry_   .emplace_back(new ospray::cpp::Geometry("streamlines")).get();
geometry->setMaterial(material);
geometry->set        ("vertex"      , *vertex);
geometry->set        ("vertex.color", *color );
geometry->set        ("index"       , *index );
geometry->set        ("radius"      , streamlines->radius);
geometry->commit     ();
model_  ->addGeometry(*geometry);

}

ospcommon::vec3f region_lower_bounds { streamlines->bounds[0][0], streamlines->bounds[0][1], streamlines->bounds[0][2] }; ospcommon::vec3f region_upperbounds { streamlines->bounds[1][0], streamlines->bounds[1][1], streamlines->bounds[1][2] }; model->set ("id" , communicator->rank()); model->set ("region.lower", region_lowerbounds ); model->set ("region.upper", region_upperbounds ); model->commit();

renderer->set ("model", *model); renderer_->commit();

framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM); } void raytracer::setcamera (const camera& camera ) { camera->set ("pos", ospcommon::vec3f{camera.position[0], camera.position[1], camera.position[2]}); camera->set ("dir", ospcommon::vec3f{camera.forward [0], camera.forward [1], camera.forward [2]}); camera->set ("up" , ospcommon::vec3f{camera.up [0], camera.up [1], camera.up [2]}); camera_->commit();

framebuffer_->clear (OSP_FB_COLOR | OSP_FB_ACCUM); } void raytracer::set_image_size (const std::array<std::uint64_t, 2>& image_size ) { framebuffersize = image_size;

camera_->set ("aspect", framebuffersize[0] / static_cast(framebuffersize[1])); camera_->commit();

framebuffer_ = std::make_unique( ospcommon::vec2i(static_cast(framebuffersize[0]), static_cast(framebuffersize[1])), OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FBACCUM); framebuffer->clear(OSP_FB_COLOR | OSP_FB_ACCUM); }

void raytracer::trace (const std::size_t iterations ) { for (std::sizet i = 0; i < iterations; ++i) renderer->renderFrame(framebuffer_, OSP_FB_COLOR | OSP_FB_ACCUM); } image raytracer::serialize () { image image; const auto bytes = static_cast<std::uint8_t>(framebuffer_->map(OSP_FB_COLOR)); image.set_data (bytes, framebuffersize[0] framebuffersize[1] sizeof(std::array<std::uint8_t, 4>)); image.mutable_size()->Resize(2, 0); image.mutable_size()->Set (0, framebuffersize[0]); image.mutable_size()->Set (1, framebuffersize[1]); framebuffer_->unmap(bytes); return image; } }

- I can open up the whole repo if you like.
- I'll get back onto this in 11 days.

[Twinklebear]

Hey @acdemiralp , the code that works on 1.7.3 looks like it should also work on 1.8, since the region bounds are set on each model. One possible issue is using the camera image start end parameters to do the y flip, this may not be handled by the compositor properly when it's determining which data touches which tile. It might be worth trying with that off, and see if the image comes out correct (though upside down). If you can share the repo that would be great, I'd be interested in seeing the code which sets up and uses the streamlines here, and if I can test it myself to see what's going on.

[will]

Hey 11 days or 111, I’m cool with whatever

On Wed, Feb 20, 2019 at 11:29 Ali Can Demiralp notifications@github.com wrote:

I decided to try out Ospray 1.7.3 just as a last hour effort at work, without any changes to code and it worked, showing me the image I expected to see: [image: test3 json nodes_2 threads_8 rank_0] https://user-images.githubusercontent.com/6049621/53118138-c3c89580-354c-11e9-892d-02156caacf63.png

Due to time constraints (EGPGV), I cannot spend more time on fixing what's wrong with the latest distributed renderer at the moment, since I still have to make that mid-line disappear and write a half-paper on top of that in 11 days.

I will get back onto it after the deadline, because your library is awesome in many ways and I wanna make it better, although it really makes me sad when behavior changes randomly version to version, resulting in 2 research paper ragequits until today.

@will https://github.com/will:

• The rank==0 was due to debugging as I mentioned earlier. It nevertheless gave the earlier results you mentioned without that if statement.
• Those are not volumes but very tiny streamlines seen from afar.
• This code works in 1.7.3 but not in 1.8.0:

include <prs/stages/raytracer.hpp>

namespace prs {raytracer::raytracer (boost::mpi::communicator communicator, const std::size_t threadcount) : communicator(communicator), rawcommunicator(communicator) { if (ospLoadModule("ispc") != OSP_NO_ERROR) throw std::runtime_error("Failed to load OSPRay ISPC module"); if (ospLoadModule("mpi") != OSP_NO_ERROR) throw std::runtime_error("Failed to load OSPRay MPI module");

const auto device = ospray::cpp::Device("mpi_distributed"); ospDeviceSetStatusFunc(device.handle(), [ ] ( const char message) { std::cout << message << "\n"; }); ospDeviceSetErrorFunc (device.handle(), [ ] (OSPError error, const char message) { std::cout << message << "\n"; }); device.set ("numThreads" , thread_count ); device.set ("setAffinity" , 1 ); device.set ("masterRank" , 0 ); device.set ("worldCommunicator", &rawcommunicator);//#if _DEBUG device.set ("logLevel" , 3 ); device.set ("logOutput" , "cout" ); device.set ("errorOutput" , "cerr" );//#endif device.commit (); device.setCurrent();

auto material = ospray::cpp::Material("scivis", "OBJMaterial"); material.set ("Ks", 0.7F, 0.7F, 0.7F); material.set ("Ns", 10.0F); material.commit();

geometry_ .emplace_back(std::make_unique("streamlines")); vertexdata.emplace_back(std::make_unique(0, OSP_FLOAT3A, nullptr)); vertexdata.back()->commit(); colordata .emplace_back(std::make_unique(0, OSP_FLOAT4 , nullptr)); colordata .back()->commit(); indexdata .emplace_back(std::make_unique(0, OSP_INT , nullptr)); indexdata .back()->commit();

geometry_[0]->set ("vertex" , vertexdata.back()); geometry_[0]->set ("vertex.color", colordata .back()); geometry_[0]->set ("index" , *indexdata .back()); geometry[0]->set ("radius" , 0.1F); geometry[0]->setMaterial(material); geometry_[0]->commit ();

model_ = std::makeunique(); model->addGeometry(*geometry[0]); model->set ("id", communicator->rank()); model->commit ();

camera_ = std::makeunique("perspective"); camera->set ("fovy" , 68.0F); camera->set ("imageStart", ospcommon::vec2f{0.0F, 1.0F}); camera->set ("imageEnd" , ospcommon::vec2f{1.0F, 0.0F}); camera_->commit();

auto ambient_light = ospray::cpp::Light("ambient"); ambient_light.set ("intensity", 0.2F); ambient_light.commit(); const auto ambient_handle = ambient_light.handle();

auto distant_light = ospray::cpp::Light("distant"); distant_light.set ("direction" , 1.0F, 1.0F, -0.5F); distant_light.set ("color" , 1.0F, 1.0F, 0.8F); distant_light.set ("intensity" , 0.8F); distant_light.set ("angularDiameter", 1.0F); distant_light.commit(); const auto distant_handle = distant_light.handle();

std::vector lights_list = {ambient_handle, distanthandle}; lights = std::make_unique(lights_list.size(), OSP_LIGHT, lightslist.data()); lights->commit();

renderer_ = std::make_unique("mpiraycast"); renderer->set ("aoSamples" , 1 ); renderer->set ("aoTransparencyEnabled", false ); renderer->set ("oneSidedLighting" , true ); renderer->set ("spp" , 1 ); renderer->set ("bgColor" , 1.0F, 1.0F, 1.0F, 1.0F); renderer->set ("camera" , *camera); renderer->set ("lights" , *lights); renderer->set ("model" , *model ); renderer_->commit();

framebuffer_ = std::make_unique(ospcommon::vec2i(32, 32), OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FBACCUM); framebuffer->clear(OSP_FB_COLOR | OSP_FBACCUM); }raytracer::~raytracer() { framebuffer.reset(); lights_ .reset(); indexdata .clear(); colordata .clear(); vertexdata.clear(); geometry .clear(); model .reset(); camera .reset(); renderer .reset(); ospShutdown(); } void raytracer::set_streamlines(streamlines<>* streamlines) { auto material = ospray::cpp::Material("scivis", "OBJMaterial"); material.set ("Ks", 0.7F, 0.7F, 0.7F); material.set ("Ns", 10.0F); material.commit();

model_ = std::makeunique(); geometry .clear(); vertexdata.clear(); colordata .clear(); indexdata .clear(); geometry_ .reserve(streamlines->chunks.size()); vertexdata.reserve(streamlines->chunks.size()); colordata .reserve(streamlines->chunks.size()); indexdata .reserve(streamlines->chunks.size());

for(auto& chunk : streamlines->chunks) { if (chunk.indices.empty()) continue;

const auto vertex   = vertex_data_.emplace_back(new ospray::cpp::Data(chunk.vertices.size(), OSP_FLOAT3A, chunk.vertices.data(), OSP_DATA_SHARED_BUFFER)).get(); vertex->commit();
const auto color    = color_data_ .emplace_back(new ospray::cpp::Data(chunk.colors  .size(), OSP_FLOAT4 , chunk.colors  .data(), OSP_DATA_SHARED_BUFFER)).get(); color ->commit();
const auto index    = index_data_ .emplace_back(new ospray::cpp::Data(chunk.indices .size(), OSP_INT    , chunk.indices .data(), OSP_DATA_SHARED_BUFFER)).get(); index ->commit();
const auto geometry = geometry_   .emplace_back(new ospray::cpp::Geometry("streamlines")).get();
geometry->setMaterial(material);
geometry->set        ("vertex"      , *vertex);
geometry->set        ("vertex.color", *color );
geometry->set        ("index"       , *index );
geometry->set        ("radius"      , streamlines->radius);
geometry->commit     ();
model_  ->addGeometry(*geometry);

}

ospcommon::vec3f region_lower_bounds { streamlines->bounds[0][0], streamlines->bounds[0][1], streamlines->bounds[0][2] }; ospcommon::vec3f region_upperbounds { streamlines->bounds[1][0], streamlines->bounds[1][1], streamlines->bounds[1][2] }; model->set ("id" , communicator->rank()); model->set ("region.lower", region_lowerbounds ); model->set ("region.upper", region_upperbounds ); model->commit();

renderer->set ("model", *model); renderer_->commit();

framebuffer_->clear(OSP_FB_COLOR | OSP_FB_ACCUM); }void raytracer::setcamera (const camera& camera ) { camera->set ("pos", ospcommon::vec3f{camera.position[0], camera.position[1], camera.position[2]}); camera->set ("dir", ospcommon::vec3f{camera.forward [0], camera.forward [1], camera.forward [2]}); camera->set ("up" , ospcommon::vec3f{camera.up [0], camera.up [1], camera.up [2]}); camera_->commit();

framebuffer_->clear (OSP_FB_COLOR | OSP_FB_ACCUM); }void raytracer::set_image_size (const std::array<std::uint64_t, 2>& image_size ) { framebuffersize = image_size;

camera_->set ("aspect", framebuffersize[0] / static_cast(framebuffersize[1])); camera_->commit();

framebuffer_ = std::make_unique( ospcommon::vec2i(static_cast(framebuffersize[0]), static_cast(framebuffersize[1])), OSP_FB_SRGBA, OSP_FB_COLOR | OSP_FBACCUM); framebuffer->clear(OSP_FB_COLOR | OSP_FB_ACCUM); } void raytracer::trace (const std::size_t iterations ) { for (std::sizet i = 0; i < iterations; ++i) renderer->renderFrame(framebuffer_, OSP_FB_COLOR | OSP_FB_ACCUM); } image raytracer::serialize () { image image; const auto bytes = static_cast<std::uint8_t>(framebuffer_->map(OSP_FB_COLOR)); image.set_data (bytes, framebuffersize[0] framebuffersize[1] sizeof(std::array<std::uint8_t, 4>)); image.mutable_size()->Resize(2, 0); image.mutable_size()->Set (0, framebuffersize[0]); image.mutable_size()->Set (1, framebuffersize[1]); framebuffer_->unmap(bytes); return image; } }

• I can open up the whole repo if you like.
• I'll get back onto this in 11 days.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ospray/ospray/issues/300#issuecomment-465720697, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAHtVvJRxQtzac851vTGdPMaqlAag77ks5vPaIAgaJpZM4bDyaP .

[Twinklebear]

@acdemiralp for a quick test I changed the MPI distributed spheres tutorial to flip the image using the camera start/end parameters and also find that it hangs. I wonder if removing these will get it rendering? Then you could flip the framebuffer in a post-process on rank 0 for EGPGV, and I'll work on resolving this bug. Let me know if that works

[Twinklebear]

I took a look at getting the image start/end camera parameters working properly in the distributed renderer, and think I should have this working now. At least, in the distributed tutorials I can change the image start/end parameters and get the cropped/flipped/etc. render I expect out.

@acdemiralp if you've got time, could you try applying the following patch to OSPRay's perspective camera? The file is ospray/camera/PerspectiveCamera.cpp

diff --git a/origin/devel:ospray/camera/PerspectiveCamera.cpp b/will/mpi-cam-subregion:ospray/camera/PerspectiveCamera.cpp
index 22e5d40..aa5a001 100644
--- a/origin/devel:ospray/camera/PerspectiveCamera.cpp
+++ b/will/mpi-cam-subregion:ospray/camera/PerspectiveCamera.cpp
@@ -116,8 +116,9 @@ namespace ospray {
     const float t = 1.0 / denom;

     const vec3f screenDir = r * t - dir_00;
-    const vec2f sp = vec2f(dot(screenDir, normalize(dir_du)),
-                           dot(screenDir, normalize(dir_dv))) / imgPlaneSize;
+    vec2f sp = vec2f(dot(screenDir, normalize(dir_du)),
+                     dot(screenDir, normalize(dir_dv))) / imgPlaneSize;
+    sp = (sp - imageStart) / (imageEnd - imageStart);
     const float depth = sign(t) * length(v);
     // TODO: Depth of field radius
     return ProjectedPoint(vec3f(sp.x, sp.y, depth), 0);

If it's not working after this or having other issues, I'd be happy to take a look at the repo and see what's going on. Though if that's the case it might be better to stick with 1.7.3 for EGPGV. I think the line across issue may be due to not having ghost streamlines across the boundary of the two processes, so streamlines that may be owned by one process but cross into the other's domain are missing, or clipped. Or if they're not being computed in the first place, it could also be due to missing the ghost voxels across the boundary from the other process (though here I'm not as familiar with the code, so mostly a guess).