Open vlordier opened 4 years ago
could you show your code for creating the TF estimator and launching the training job as well?
creating the TF estimator
tf_estimator = TensorFlow( entry_point = train.py,
train_use_spot_instances = False,
train_max_wait = 36000,
train_max_run= 36000,
role = role,
train_instance_count = 1,
train_instance_type = 'ml.p2.8xlarge',
framework_version = '2.0.0',
py_version = 'py3',
input_mode = 'Pipe',
source_dir = source_dir,
hyperparameters = hyperparameters,
distributions = distributions,
)
with distributions as
distributions={
'mpi': {
'enabled': True,
'processes_per_host': 8,
'custom_mpi_options': '--NCCL_DEBUG INFO'
}
}
since I am looking to
Launching the training job with
tf_estimator.fit(inputs=data_channels, logs=True)
with data_channels as
data_channels = {'training': train_data, 'testing' : test_data, 'validation' : val_datal}
and train_data
defined as
train_data = sagemaker.session.s3_input(augmented_manifest_file_on_s3,
distribution = 'FullyReplicated',
content_type = 'image/jpeg',
s3_data_type = 'AugmentedManifestFile',
attribute_names = ['image-ref', 'label'],
input_mode = 'Pipe',
record_wrapping = 'RecordIO')
This notebook here is a good starting point to put together a little demo of how to use Pipe Mode on Augmented Manifest Files
Have you seen the documentation we have on using augmented manifest files with PipeModeDataset? https://github.com/aws/sagemaker-tensorflow-extensions#using-the-pipemodedataset-with-sagemaker-augmented-manifest-files https://docs.aws.amazon.com/sagemaker/latest/dg/augmented-manifest.html
Marking it with a 'feature request' label for the notebook with a better explanation.
I am also encountering the above issue when trying to decode PNG images.
OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Could not parse example input
Have we got any idea what is causing the above issue?
@vlordier Update for you if you have not found the answer, I removed the features and fed in the example1 / 2 in your case directly although you will need to tf.strings.to_number() on your label. This then worked for me
It would really help to have a full end to end example of, say, image classification with augmentedManifestFile + PipeModeDataset
as I keep getting errors of formats like
tensorflow/core/framework/op_kernel.cc:1622] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Could not parse example input, value: '����
I build a jsonl augumentedManifest with
then preparing training channel as
and launching the
.fit
asin my entry script, I have
and as a modified example parser, I have
`def example_parser(exemple1, exemple2):
`
What am I doing wrong ? The documentation here is not clear about using augmented manifest files