SoroushMehraban
commented
11 months ago You don't need .avi files for finetuning. I recommend doing it this way:
- Write an script that converts csv files into numpy format. The numpy shape for each 2D/3D sequence should be (T, 17, 3) where T is number of frames in the whole sequence, 17 is the number of joints in Human3.6M format (See
H36M_JOINT_TO_LABELhere for seeing the orders) and 3 is (x, y, conf_score) for 2D sequence and (x, y, z) for 3D sequence. Challenges you might face:- You might not have confidence scores for 2D keypoints -> In that case set them to be 1.
- Yo might not have the exact joints. Like you might have left hip and right hip but not the center hip -> Try to estimate the missing joint by taking their mean (Or other estimation that makes more sense to you)
- After finishing the first step, it is expected to have a directory (let's call it
keypoints) with the following structure:. └── keypoints/ ├── train/ │ ├── sequence_1_2D.npy │ ├── sequence_1_3D.npy │ ├── sequence_2_2D.npy │ ├── sequence_2_3D.npy │ └── ... └── val/ ├── sequence_1_2D.npy ├── sequence_1_3D.npy ├── sequence_2_2D.npy ├── sequence_2_3D.npy └── ...Then you need to create a torch dataset for them. In the dataset
__init__function you can do the following:- The
__init__receives path tokeypointsand data_split (either 'train' or 'val'). Then you list all the numpy files in train or validation directory - Next, you can read them all and store them in a dictionary where keys are the sequence name and values are the actual numpy ndarray.
- Now the issue is that some sequences are more than 243 frames and some are shorter. So you need to split them into clips:
- Sequences that are shorter than 243 frames can be extrapolated to become 243 frames.
- This is doable simply by repeating the adjacent frames. Like instead of being 1, 2, 3, 4, ..., 80. Now it becomes 1, 1, 1, 2, 2, 2, 3, 3, 3 , ..., 80, 80, 80 such that total number of frames are 243 now.
- Sequences that are longer than 243 frames can be divided into clips of 243 frames each.
- The
- After finishing the first step, it is expected to have a directory (let's call it
- A trick here that used in our project is to use concept of stride, that is set to 81. So instead of clip one to be from frame 0 to 242 and clip two to be from frame 243 to 485, now clip one is from 0 to 242 but second clip is from 81 to 323.
- Note that by doing this we make the training data more than what it truly is and it doesn't make sense to do it in validation data. So in validation data stride should be 243 (no overlapping between sequences).
- So by turning video sequences into clips, now the dictionary instead of being keys as sequence name and values as the whole sequence, now the value is list of clips.
- Finally you can turn this dictionary into a large list. So eventually you'll have a large list for 2D keypoints and a list with exact same size for 3D keypoints.
- In the
__getitem__method of your dataset you simply receive an index from torch dataloader and retrieve it from those big lists you defined before. You can also do mirroring augmentation with probability of 50% (same as what we did here)
Note: You have to normalize both 2D and 3D sequences for the training to be in value [-1, 1]. This helps the model to have stable output. This is the function used for MotionAGFormer normalization:
def normalize(keypoints, w, h, is_3d=False):
result = np.copy(keypoints)
result[..., :2] = keypoints[..., :2] / w * 2 - [1, h / w] # for width and height
if is_3d:
result[..., 2:] = keypoints[..., 2:] / w * 2 # for depth in 3D keypoints
return resultwhere w and h are width and height of the video that you can get it from those avi files.
Note: In the evaluation code you have to report MPJPE in millimeters. But after normalization the scale is different and is in range [-1, 1]. So you have to denormalize it back to the millimeters. For doing it, in your torch dataset you can define the following method:
def denormalize(self, keypoints, idx, is_3d=False):
h, w = self.data_list_camera[idx]
result = np.copy(keypoints)
result[..., :2] = (keypoints[..., :2] + np.array([1, h / w])) * w / 2
if is_3d:
result[..., 2:] = keypoints[..., 2:] * w / 2
return resultNote that I also assumed when you clip the videos into sequences of 243 frames, you also store a variable called data_list_camera that stores width and height of each clip in the avi video (in case that different videos are captured with different camera formats).
The good news is, for one my course projects, I already did these things so you can use them with minor modifications. In your __init__ method of your class assuming that you store the 2D keypoints dictionary in data_2d and 3D keypoints dictionary in data_3d, and if n_frames=243 and stride=81, we have:
def __init__(self, keypoints_path, data_split, n_frames=243, stride=81, ...):
...
self.data_list_2d, self.data_list_3d, self.data_list_camera = self.split_into_clips(data_2d, data_3d, n_frames, stride)
assert len(self.data_list_2d) == len(self.data_list_3d)
assert len(self.data_list_2d) == len(self.data_list_camera)where split_into_clips is defined as follows:
def split_into_clips(self, data_2d, data_3d, n_frames, stride):
data_list_2d, data_list_3d, data_list_camera = [], [], []
for sequence_name in data_2d:
keypoints_2d = data_2d[sequence_name]
keypoints_3d = data_3d[sequence_name]['keypoints']
res_h = data_3d[sequence_name]['res_h']
res_w = data_3d[sequence_name]['res_w']
keypoints_2d = self.normalize(keypoints_2d, res_w, res_h)
keypoints_3d = self.normalize(keypoints_3d, res_w, res_h, is_3d=True)
keypoints_2d = keypoints_2d[:keypoints_3d.shape[0]] # Make sure the lengths are equal
clips_2d, clips_3d = self.partition(keypoints_2d, keypoints_3d, n_frames, stride)
data_list_2d.extend(clips_2d)
data_list_3d.extend(clips_3d)
data_list_camera.extend([(res_h, res_w)] * len(clips_2d))
return data_list_2d, data_list_3d, data_list_cameraNote that it is expected that data_3d to have the following format:
{
'<SEQUENCE_NAME>': {
"keypoints": <NUMPY ARRAY>,
"res_h": <HEIGHT OF THE CLIP>,
"res_w" <WIDTH OF THE CLIP>
}
}Then parition method is defined as:
def partition(self, keypoints_2d, keypoints_3d, clip_length, stride):
if self.data_split == "test":
stride = clip_length
clips_2d, clips_3d = [], []
video_length = keypoints_2d.shape[0]
if video_length <= clip_length:
new_indices = self.resample(video_length, clip_length)
clips_2d.append(keypoints_2d[new_indices])
clips_3d.append(keypoints_3d[new_indices])
else:
start_frame = 0
while (video_length - start_frame) >= clip_length:
clips_2d.append(keypoints_2d[start_frame:start_frame + clip_length])
clips_3d.append(keypoints_3d[start_frame:start_frame + clip_length])
start_frame += stride
new_indices = self.resample(video_length - start_frame, clip_length) + start_frame
clips_2d.append(keypoints_2d[new_indices])
clips_3d.append(keypoints_3d[new_indices])
return clips_2d, clips_3dAnd resample method, responsible for extrapolation, is defined as:
@staticmethod
def resample(original_length, target_length):
"""
Adapted from https://github.com/Walter0807/MotionBERT/blob/main/lib/utils/utils_data.py#L68
Returns an array that has indices of frames. elements of array are in range (0, original_length -1) and
we have target_len numbers (So it interpolates the frames)
"""
even = np.linspace(0, original_length, num=target_length, endpoint=False)
result = np.floor(even)
result = np.clip(result, a_min=0, a_max=original_length - 1).astype(np.uint32)
return resultFinal note: You can change the evaluate function throughout training to work this way:
def evaluate(args, model, test_loader, device):
print("[INFO] Evaluation")
model.eval()
mpjpe_all, p_mpjpe_all = AverageMeter(), AverageMeter()
with torch.no_grad():
for x, y, indices in tqdm(test_loader):
batch_size = x.shape[0]
x = x.to(device)
if args.flip:
batch_input_flip = flip_data(x)
predicted_3d_pos_1 = model(x)
predicted_3d_pos_flip = model(batch_input_flip)
predicted_3d_pos_2 = flip_data(predicted_3d_pos_flip) # Flip back
predicted_3d_pos = (predicted_3d_pos_1 + predicted_3d_pos_2) / 2
else:
predicted_3d_pos = model(x)
if args.root_rel:
predicted_3d_pos[:, :, 0, :] = 0 # [N,T,17,3]
else:
y[:, 0, 0, 2] = 0
predicted_3d_pos = predicted_3d_pos.detach().cpu().numpy()
y = y.cpu().numpy()
denormalized_predictions = []
for i, prediction in enumerate(predicted_3d_pos):
prediction = test_loader.dataset.denormalize(prediction,
indices[i].item(), is_3d=True)
denormalized_predictions.append(prediction[None, ...])
denormalized_predictions = np.concatenate(denormalized_predictions)
# Root-relative Errors
predicted_3d_pos = denormalized_predictions - denormalized_predictions[..., 0:1, :]
y = y - y[..., 0:1, :]
mpjpe = calculate_mpjpe(predicted_3d_pos, y)
p_mpjpe = calculate_p_mpjpe(predicted_3d_pos, y)
mpjpe_all.update(mpjpe, batch_size)
p_mpjpe_all.update(p_mpjpe, batch_size)
print(f"Protocol #1 error (MPJPE): {mpjpe_all.avg} mm")
print(f"Protocol #2 error (P-MPJPE): {p_mpjpe_all.avg} mm")
return mpjpe_all.avg, p_mpjpe_all.avgNote that in function above:
- It is expected that for test dataset, the values it returns in
__getitem__is already denormalized - Also it returns the index (that
__getitem__receives) in addition to the 2D and 3D sequence.
Hope it helps! Cheers.
elisha0904
Millba
I want to use the data I have for finetuning, but I'm not sure how to preprocess it in the right format. The types of data I have are as follows:
Video: .avi format videos filmed from 8 different directions for a single movement (8 files per movement). Image: A folder of images in .jpg format, cut frame by frame from the video. 2D Keypoint: CSV files storing frame-by-frame 2D keypoint values for each video (8 files per movement). 3D Keypoint: CSV file storing frame-by-frame 3D keypoint values for a single movement (1 file per movement). Annotation: JSON files containing information about each video.
training/ └── raw_data/ └── video/ └── CA01_1_camera0/ ├── Motion2-1.avi └── image/ └── CA01_1_camera0/ ├── 0.jpg ├── 1.jpg └── labeling_data/ └── 2d_keypoint/ ├── Motion2-1.csv └── 3d_keypoint/ ├── CA01_1.csv ├── CA01_2.csv └── annotation/ ├── CA01_1_camera0.json ├── CA01_1_camera1.json