pytorch

[eric-yyjau]

Questions

• How to use torch.auto_grad Variable?

[eric-yyjau]

Mask an array

• Use the operate of the array

  a = torch.tensor(np.arange(10))
  # tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
  b = a>5
  # tensor([0, 0, 0, 0, 0, 0, 1, 1, 1, 1], dtype=torch.uint8)
  a[b]
  # tensor([6, 7, 8, 9])

• Use another array

  b = torch.tensor([0, 0, 0, 0, 0, 0, 1, 1, 1, 1])
  # tensor([0, 0, 0, 0, 0, 0, 1, 1, 1, 1])
  a[b]
  # tensor([0, 0, 0, 0, 0, 0, 1, 1, 1, 1]) --> Note! The result is different from the previous one.
  a[b.to(torch.uint8)]
  # tensor([6, 7, 8, 9]) --> same results
  a[b.nonzero().squeeze()]
  # tensor([6, 7, 8, 9]) --> same results

[eric-yyjau]

Dataset

MNist

• an example of pytorch on mnist dataset https://gist.github.com/xmfbit/b27cdbff68870418bdb8cefa86a2d558

[eric-yyjau]

Model manipulation

• mortezamg63/Accessing-and-modifying-different-layers-of-a-pretrained-model-in-pytorch https://github.com/mortezamg63/Accessing-and-modifying-different-layers-of-a-pretrained-model-in-pytorch

[eric-yyjau]

Reproducibility

Ref

• Reproducibility — PyTorch master documentation https://pytorch.org/docs/stable/notes/randomness.html

Q: Data loader: shuffle = True

• can't reproduce results even set all random seeds · Issue #7068 · pytorch/pytorch https://github.com/pytorch/pytorch/issues/7068

Q: Data loader: num_workers copy all things?

• torch.utils.data — PyTorch master documentation https://pytorch.org/docs/stable/data.html

  def worker_init_fn(worker_id):
  """The function is designed for pytorch multi-process dataloader.
  Note that we use the pytorch random generator to generate a base_seed.
  Please try to be consistent.
  References:
     https://pytorch.org/docs/stable/notes/faq.html#dataloader-workers-random-seed
  """
  base_seed = torch.IntTensor(1).random_().item()
  # print(worker_id, base_seed)
  np.random.seed(base_seed + worker_id)

[eric-yyjau]

Check gradient

Check gradient flow in network - PyTorch Forums https://discuss.pytorch.org/t/check-gradient-flow-in-network/15063/10

[eric-yyjau]

Cuda

runtime error

• check nans "invalid configuration argument " error for the call of CUDA kernel? - Stack Overflow https://stackoverflow.com/questions/16125389/invalid-configuration-argument-error-for-the-call-of-cuda-kernel