andrevitorelli
commented
3 years ago I'm toying with the idea of modifying the tfds class to not just unpack and assign images, but to generate them. In this way we can have just a text file as a "catalogue" (eg, with position, ellipticity, SNR) and have galsim generate images as needed. If this is too hard of course I could have a separate class do this (which is what I'm working on right now). For reference, the tfds base class structure:
class galgen(tfds.core.GeneratorBasedBuilder):
"""DatasetBuilder for galgen dataset."""
VERSION = tfds.core.Version('1.0.0')
RELEASE_NOTES = {
'0.0.0': 'Initial release.',
}
def _info(self) -> tfds.core.DatasetInfo:
"""Dataset metadata (homepage, citation,...)."""
return tfds.core.DatasetInfo(
builder=self,
features=tfds.features.FeaturesDict({
'image': tfds.features.Image(shape=(256, 256, 3)),
'label': tfds.features.ClassLabel(names=['no', 'yes']),
}),
)
def _split_generators(self, dl_manager: tfds.download.DownloadManager):
"""Download the data and define splits."""
extracted_path = dl_manager.download_and_extract('http://data.org/data.zip')
# dl_manager returns pathlib-like objects with `path.read_text()`,
# `path.iterdir()`,...
return {
'train': self._generate_examples(path=extracted_path / 'train_images'),
'test': self._generate_examples(path=extracted_path / 'test_images'),
}
def _generate_examples(self, path) -> Iterator[Tuple[Key, Example]]:
"""Generator of examples for each split."""
for img_path in path.glob('*.jpeg'):
# Yields (key, example)
yield img_path.name, {
'image': img_path,
'label': 'yes' if img_path.name.startswith('yes_') else 'no',
}
EiffL
To run the various tests we will be interested in, we will need some ready available galaxy images, under a series of noise and PSF conditions. This issue is to track the development of a TensorFlow Dataset class that uses GalSim to generate a dataset of postage stamps, with each record having the following entries:
This way, we will be able to add a random shear to the galaxy image on the fly, and add whatever amount of noise we want.
Ultimately, we also probably want to be able to load sims made by Axel. @aguinot can you describe here where to find your sims and how to read them?