patchkernel: improve adaption/partitioning tracking

[andrea-iob]

Adaption/partitioning tracking is extended also to vertices and interfaces. However, tracking of vertices and interfaces offers only a limited set of information.

During adaption, information available for tracking purposes are the following:

• internal cells that have been coarsened/refined;
• new internal vertices that have been created;
• internal vertices that have been deleted;
• new ghost cells that have been created;
• new ghost vertices that have been created;
• ghost cells that have been deleted;
• ghost vertices that have been deleted;
• new interfaces that have been created;
• interfaces that have been deleted.

During partitioning, information available on the sender side for tracking purposes are the following:

• internal cells that have been sent;
• internal vertices that have been sent;
• new ghost cells that have been created (some of the internal cells that have been sent may have become ghosts cells);
• new ghost vertices that have been created (some of the internal vertices that have been sent may have become ghosts vertices);
• ghost cells that have been deleted;
• ghost vertices that have been deleted;
• new interfaces that have been created;
• interfaces that have been deleted.

During partitioning, information available on the receiver side for tracking purposes are the following:

• internal cells that have been received;
• internal vertices that have been received;
• new ghost cells that have been created;
• new ghost vertices that have been created;
• ghost cells that have been deleted (some ghost cells may have been replaced by internal cells that have just been received);
• ghost vertices that have been deleted (some ghost vertices may have been replaced by internal vertices that have just been received);
• new interfaces that have been created;
• interfaces that have been deleted.

[andrea-iob]

Test failures seems related to problems with travis; pull requests #371 and #372 contain also these changes and there the tests pass just fine.

[edoardolombardi]

The pull seems not working for vertices tracking. The adaption info returned for entity VERTEX are empty.

You can see with following modifications to test_volunstructured_parallel_00001 as following (this code chunk at the place of line 523):

` std::vector adaptionInfo = patch_3D->partition(cellRanks, true, true);

log::cout() << "adaption info size: " << adaptionInfo.size() << std::endl;

for (auto & info : adaptionInfo) {
    if (info.entity == bitpit::adaption::Entity::ENTITY_CELL) {
        log::cout() << "cell info previous: " << info.previous << std::endl;
        log::cout() << "cell info current: " << info.current << std::endl;
    }
    else if (info.entity == bitpit::adaption::Entity::ENTITY_VERTEX) {
        log::cout() << "vertex info previous: " << info.previous << std::endl;
        log::cout() << "vertex info current: " << info.current << std::endl;
    }

}

` The previous/current vectors in case of ENTITY_VERTEX adaption entity are empty. But maybe I'm doing something wrong in the usage...

@andrea-iob could you add in the pull a test or an example of usage of the tracking results, please? For both purposes, to test the implementation (at least that there are no empty structures neither segmentation fault, or better if a predefined partitioning is reached) and to help the usage.

I don't know where is the best place and the best way to explain it, but could you also add some comments to the documentation or of the adaption info structures (completely missing) or to the partition methods to explain what the returning objects mean, please? What are the current ids, the previous ones for instance, where the user can find info on ghost cells/vertices/interfaces, etc. Practically, I think that it should be enough explaining where each one of the info that you listed in the description of the pull can be retrieved.

Thanks

[andrea-iob]

Tracking vertex send/receive during partitioning is more tricky than expected. The image represent a 2x2 mesh before (left) and after (right) the bottom left cell has been send from process #0 to process #2.

Cell exchange involves ranks #⁠0 and #⁠2, but after the cell have been send there are some vertices that are now owned by process #⁠1 (the owner of a vertex is the process with the lowest rank). The tracking should than report that those vertices have been sent to process #⁠1 from process #⁠0. I'm not sure how I can get that information in the partitioning prepare stage (this is when vertex data to be send should be collected).

The best solution would be to void tracking data exchange for vertices and relay on cell exchange to communicate vertex data.

        for (const auto &adaptionInfo : updateData) {
            if (adaptionInfo.entity != adaption::ENTITY_CELL) {
                continue;
            }

            int rank = adaptionInfo.rank;
            adaption::Type adaptionType = adaptionInfo.type;
            if (adaptionType == adaption::TYPE_PARTITION_SEND) {
                const std::vector<long> &cellsToSend = adaptionInfo.previous;
                std::vector<long> verticesToSend = patch->getOrderedCellsVertices(cellsToSend, true, true);

                // Start vertex communications
                ...              
            }
        }

I've already modified the branch to allow that, after some more tests I'll push it.