This project improves the usability of the sample code from WWDC20 session 10611: Explore ARKit 4. Note that the sample code is also on the original
branch and the original code from WWDC20 can be checked out at the first commit. The original project places points in the real-world using the scene's depth data to visualize the shape of the physical environment.
Check out the walkthrough video on Youtube.
This project adds the following functions:
Add buttons to start/pause recordings.
Save ARFrame raw data asynchronously at a chosen rate when recording. The selected data include
// custom struct for pulling necessary data from arframes
struct ARFrameDataPack {
var timestamp: Double
var cameraTransform: simd_float4x4
var cameraEulerAngles: simd_float3
var depthMap: CVPixelBuffer
var smoothedDepthMap: CVPixelBuffer
var confidenceMap: CVPixelBuffer
var capturedImage: CVPixelBuffer
var localToWorld: simd_float4x4
var cameraIntrinsicsInversed: simd_float3x3
}
The sampling rate is controlled by a slider. One of every n
new frames will be saved. The current sampling rate will be indicated in the filename, i.e. {timestamp}_{samplingRate}.[json|jpeg]
. Please note that all frames will contribute to the point cloud as well as AR display. In other words, all the points in AR will be saved to PLY
point cloud file but only 1/n
of them will be saved as json
and jpeg
.
The captured images are stored in jpeg
format and others are coded into json
files of which the format is specified as below.
struct DataPack: Codable {
var timestamp: Double
var cameraTransform: simd_float4x4 // The position and orientation of the camera in world coordinate space.
var cameraEulerAngles: simd_float3 // The orientation of the camera, expressed as roll, pitch, and yaw values.
var depthMap: [[Float32]]
var smoothedDepthMap: [[Float32]]
var confidenceMap: [[UInt8]]
var localToWorld: simd_float4x4
var cameraIntrinsicsInversed: simd_float3x3
}
They can be retrieved in Finder with USB connection. Those raw data make it possible to leverage photogrammetry techniques for various tasks.
Save the point cloud in PLY
format when the recording is stopped.
Add low memory warning and file saving progress.
This answer has been very helpful in exporting point cloud in PLY format.
According to this answer, releasing ARFrame memory pool holding is important. If ARFrame currentFrame
is passed into time-consuming async tasks like converting data formats or saving files to disks, the memory pool used by ARFrame is retained and no more frames can be written to the pool. In this case, a warning will arise.
ARSessionDelegate is retaining XX ARFrames.
To solve this problem, copy over the selected data to custom struct and pass that to async tasks. To deep copy CVPixelBuffer
, try this code.
To get warning on low memory, see this doc.
Similar projects