Dataset Not Available on LSMDC?

[dipikakhullar]

Hi @JiwanChung , I downloaded the task 1 dataset from LSMDC, and the files the codebase are looking for are no longer there. Here I have attached the files I have after downloading the data from the LSMDC webpage

Here is a screenshot of all the files I have attached below. Seems like this codebase requires this file LSMDC16_annos_training_someone.csv to exist within task 1? We have 'LSMDC16_annos_val_someone.csv', 'LSMDC16_annos_test_someone.csv', 'LSMDC16_annos_training_someone.csv', 'LSMDC16_annos_blindtest.csv' in our data/lsmdc/task1 directory.

If another file or dataset is being used, would you please be able to email me (dkhullar98@berkeley.edu) or attach it here?

[JiwanChung]

I think you need 'LSMDC16_annos_training_val.csv' file, which is simply a concatenation of 'LSMDC16_annos_training_someone.csv' and 'LSMDC16_annos_val_someone.csv' files. Or you can modify 'train_path' value in 'config.py' file to 'LSMDC16_annos_training_someone.csv' to train the model on the train set only.

Originally for our LSMDC submission we used two-stage process for model selection.

1. use train set for training and val set for hyperparameter search
2. train with the found hyperparameters on the train+val set

Let me know if this solves your question.

[dipikakhullar]

Hi, if I modify the 'train path' in config.py to 'LSMDC16_annos_training_someone.csv' I see a key error. I also see a key error using the val csv. I've added some of the traceback below.

    data, _ = load_task1_with_features(path)
  File "/home/dkhullar/CharacterGrounding/tapm/code/data/task_loaders.py", line 182, in load_task1_with_features
    return {k: {**v, **all_feature[k]} for k, v in data.items()}, {}
  File "/home/dkhullar/CharacterGrounding/tapm/code/data/task_loaders.py", line 182, in <dictcomp>
    return {k: {**v, **all_feature[k]} for k, v in data.items()}, {}
KeyError: '3001_21_JUMP_STREET_00.02.55.644-00.02.56.718'

Finally, if I try concatenating both files together as you suggested with the following code. I see this error (attached below). Not sure how to proceed, would greatly appreciate all your help.

df1 = pd.read_csv("/data/lsmdc/task1/LSMDC16_annos_val_someone.csv", sep='\t')
df2 = pd.read_csv("/data/lsmdc/task1/LSMDC16_annos_training_someone.csv", sep='\t')

df1_mat = df1.as_matrix()
df2_mat = df2.as_matrix()
np_combined_data = numpy.concatenate([df1_mat, df2_mat], axis=0)
df_combined = pd.DataFrame(np_combined_data)
print(df_combined.head())
print(list(df_combined.columns))
print("LEN: ", len(df1), len(df2), len(df_combined))

df_combined.to_csv("LSMDC16_annos_train_val_someone.csv",  sep='\t')

[dipikakhullar]

HI @JiwanChung I think some of the features or feature names might be off, could you confirm this? If so, what is the fix?

[JiwanChung]

Hi Dipika Khullar,

I went on a further investigation on the issue and suspect the following problems.

1. Do you have the i3d2 folder in your feature directory? If so, please rename it to i3d_rgb or update feature_name_map.i3d_rgb key in config.py file to i3d2.
2. Again, if you have the resnet152_3 folder, rename it to resnet.

The features we used for LSMDC submissions are not the same as the officially provided features. We extracted our features with more recent variants of i3d and resnet extractors. Hence occurs the difference in names. Still, as the structure of the files should be the same, renaming the feature folders would work ok if allowing for a bit of performance decrease.

Let me know if this solves your problem. Sincerely, Jiwan Chung

On Fri, Dec 10, 2021 at 3:53 AM Dipika Khullar @.***> wrote:

HI @JiwanChung https://github.com/JiwanChung I think some of the features or feature names might be off, could you confirm this? If so, what is the fix?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JiwanChung/tapm/issues/14#issuecomment-990130049, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE36FWHMYAXMR7F2H2WEVJ3UQD3JPANCNFSM5JVUFJZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[dipikakhullar]

Hi @JiwanChung , I renamed i3d2 to i3d_rgb and resnet152_3 to resnet. I'm still having the same issue. Thank you agai for all your help, I appreciate you taking the time to help me. Is there anything else that requires renaming? Here is the full traceback:

Traceback (most recent calls WITHOUT Sacred internals):
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function
    return lazy_capture(*args, **kwargs)
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in __call__
    return self.lazy_fn(*args, **kwargs)
  File "cli.py", line 50, in train
    all_args = prepare()
  File "cli.py", line 33, in prepare
    model, tokenizer, ckpt, datasets, epoch = prepare_model()
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function
    return lazy_capture(*args, **kwargs)
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in __call__
    return self.lazy_fn(*args, **kwargs)
  File "cli.py", line 27, in prepare_model
    return get_model_ckpt(model_name)
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function
    return lazy_capture(*args, **kwargs)
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in __call__
    return self.lazy_fn(*args, **kwargs)
  File "/home/dkhullar/CharacterGrounding/tapm/code/ckpt.py", line 63, in get_model_ckpt
    datasets = get_datasets(data_path, pretrain_path)
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function
    return lazy_capture(*args, **kwargs)
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in __call__
    return self.lazy_fn(*args, **kwargs)
  File "/home/dkhullar/CharacterGrounding/tapm/code/data/dataloader.py", line 72, in get_datasets
    datasets[k] = Dataset(p)
  File "/home/dkhullar/CharacterGrounding/tapm/code/data/dataloader.py", line 18, in __init__
    self.task, self.path = load_tasks(data_path)
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function
    return lazy_capture(*args, **kwargs)
  File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in __call__
    return self.lazy_fn(*args, **kwargs)
  File "/home/dkhullar/CharacterGrounding/tapm/code/data/task_loaders.py", line 30, in load_tasks
    f = getattr(sys.modules[__name__], f"load_{task_name}")
AttributeError: module 'data.task_loaders' has no attribute 'load_tapm'

[JiwanChung]

Hi!

I created a new environment from the sketch and could not replicate your issue. Though I suspect one possibility. The data paths has to end with .../task1/LSMDC16_....csv (task1 is the important part), which I think your data paths do not fulfill. Or you can specify task_name=task1 in config.py file.

Also as a tip, adding debug=True flag helps faster debugging :)

Sincerely, Jiwan Chung

On Fri, Dec 10, 2021 at 2:18 PM Dipika Khullar @.***> wrote:

Hi @JiwanChung https://github.com/JiwanChung , I renamed i3d2 to i3d_rgb and resnet152_3 to resnet. I'm still having the same issue. Here is the full traceback:

Traceback (most recent calls WITHOUT Sacred internals): File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function return lazy_capture(*args, kwargs) File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in call return self.lazy_fn(*args, *kwargs) File "cli.py", line 50, in train all_args = prepare() File "cli.py", line 33, in prepare model, tokenizer, ckpt, datasets, epoch = prepare_model() File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function return lazy_capture(args, kwargs) File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in call return self.lazy_fn(*args, kwargs) File "cli.py", line 27, in prepare_model return get_model_ckpt(model_name) File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function return lazy_capture(*args, *kwargs) File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in call return self.lazy_fn(args, kwargs) File "/home/dkhullar/CharacterGrounding/tapm/code/ckpt.py", line 63, in get_model_ckpt datasets = get_datasets(data_path, pretrain_path) File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function return lazy_capture(*args, kwargs) File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in call return self.lazy_fn(*args, *kwargs) File "/home/dkhullar/CharacterGrounding/tapm/code/data/dataloader.py", line 72, in get_datasets datasets[k] = Dataset(p) File "/home/dkhullar/CharacterGrounding/tapm/code/data/dataloader.py", line 18, in init self.task, self.path = load_tasks(data_path) File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 51, in captured_function return lazy_capture(args, kwargs) File "/home/dkhullar/CharacterGrounding/tapm/code/exp.py", line 40, in call return self.lazy_fn(*args, **kwargs) File "/home/dkhullar/CharacterGrounding/tapm/code/data/task_loaders.py", line 30, in loadtasks f = getattr(sys.modules[name], f"load{task_name}") AttributeError: module 'data.task_loaders' has no attribute 'load_tapm'

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JiwanChung/tapm/issues/14#issuecomment-990626190, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE36FWBBSEDNEM2SQTME6F3UQGESXANCNFSM5JVUFJZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[dipikakhullar]

Hi @JiwanChung, I have fixed the data paths so that they match the instructions in your repo.

tapm
     /data
          /lsmdc
               /task1
                    LSMDC16_annos_blindtest.csv
                    LSMDC16_annos_training_someone.csv   
                    LSMDC16_annos_val_someone.csv
                    LSMDC16_annos_test_someone.csv  
                    LSMDC16_annos_train_val_someone.csv (_our train + val appended data as suggested in a comment above_)
               /features
                    /resnet
                    /i3d_rgb

However when I run

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']"

or even

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']" debug=True

I get the following error. Do you have any suggestions to debug? Do we still think this is because of how our data is set up?:

[JiwanChung]

Hi!

It seems like you are out of the available disk space. https://coderedirect.com/questions/256358/no-usable-temporary-directory-found

Sincerely, Jiwan Chung

On Thu, Dec 16, 2021 at 2:21 AM Dipika Khullar @.***> wrote:

Hi Jiwan. I have fixed the data paths so that they match the instructions in your repo.

tapm /data /lsmdc /task1 LSMDC16_annos_blindtest.csv LSMDC16_annos_training_someone.csv LSMDC16_annos_val_someone.csv LSMDC16_annos_test_someone.csv LSMDC16_annos_train_val_someone.csv (our train + val appended data as suggested in a comment above) /features /resnet /i3d_rgb

However when I run

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']"

or even

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']" debug=True

I get the following error: [image: image] https://user-images.githubusercontent.com/33533412/146234176-c977623e-15a5-44cb-a5b7-3303814d84c7.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JiwanChung/tapm/issues/14#issuecomment-995001616, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE36FWBRSGUGWWSPXN5BEXTURDFB3ANCNFSM5JVUFJZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[dipikakhullar]

Hi Jiwan, I wanted to ask you if you could provide me with a screenshot of what your data looks like. I’m running into new issues and I think it’s because the data is formatted differently.

Best, Dipika Khullar

On Dec 16, 2021, at 8:00 AM, Jiwan Chung @.***> wrote:

 Hi!

It seems like you are out of the available disk space. https://coderedirect.com/questions/256358/no-usable-temporary-directory-found

Sincerely, Jiwan Chung

On Thu, Dec 16, 2021 at 2:21 AM Dipika Khullar @.***> wrote:

Hi Jiwan. I have fixed the data paths so that they match the instructions in your repo.

tapm /data /lsmdc /task1 LSMDC16_annos_blindtest.csv LSMDC16_annos_training_someone.csv LSMDC16_annos_val_someone.csv LSMDC16_annos_test_someone.csv LSMDC16_annos_train_val_someone.csv (our train + val appended data as suggested in a comment above) /features /resnet /i3d_rgb

However when I run

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']"

or even

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']" debug=True

I get the following error: [image: image] https://user-images.githubusercontent.com/33533412/146234176-c977623e-15a5-44cb-a5b7-3303814d84c7.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JiwanChung/tapm/issues/14#issuecomment-995001616, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE36FWBRSGUGWWSPXN5BEXTURDFB3ANCNFSM5JVUFJZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

[JiwanChung]

Hi!

I presume you are referencing LSMDC16annos*_someone.csv files. We use files downloaded from the LSMDC website as-is without any modification to the file structures. If you could provide me with more details on the issues you are having (e.g. error logs, etc.), I might be able to pinpoint the problem.

Sincerely, Jiwan Chung

On Fri, Jan 7, 2022 at 5:24 AM Dipika Khullar @.***> wrote:

Hi Jiwan, I wanted to ask you if you could provide me with a screenshot of what your data looks like. I’m running into new issues and I think it’s because the data is formatted differently.

Best, Dipika Khullar

On Dec 16, 2021, at 8:00 AM, Jiwan Chung @.***> wrote:

 Hi!

It seems like you are out of the available disk space.

https://coderedirect.com/questions/256358/no-usable-temporary-directory-found

Sincerely, Jiwan Chung

On Thu, Dec 16, 2021 at 2:21 AM Dipika Khullar @.***> wrote:

Hi Jiwan. I have fixed the data paths so that they match the instructions in your repo.

tapm /data /lsmdc /task1 LSMDC16_annos_blindtest.csv LSMDC16_annos_training_someone.csv LSMDC16_annos_val_someone.csv LSMDC16_annos_test_someone.csv LSMDC16_annos_train_val_someone.csv (our train + val appended data as suggested in a comment above) /features /resnet /i3d_rgb

However when I run

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']"

or even

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']" debug=True

I get the following error: [image: image] < https://user-images.githubusercontent.com/33533412/146234176-c977623e-15a5-44cb-a5b7-3303814d84c7.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JiwanChung/tapm/issues/14#issuecomment-995001616, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AE36FWBRSGUGWWSPXN5BEXTURDFB3ANCNFSM5JVUFJZA

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you authored the thread.

— Reply to this email directly, view it on GitHub https://github.com/JiwanChung/tapm/issues/14#issuecomment-1006910185, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE36FWDNJDJODB3IDOIKRYLUUX3BFANCNFSM5JVUFJZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[dipikakhullar]

Hi @JiwanChung Happy New Year! Sorry to keep bothering you again and again. Yes, you are correct, I am referencing *LSMDC16annos_someone.csv** files.

I have a LSMDC16_annos_train_val_someone.csv file, which has the training and validation data as we discussed above. While creating this new csv, an index column was added by default. Here is a screenshot of the data:

I kept running into key errors as I mentioned above. To rectify such errors, I tried to modify the keys variable in the following functions:

code.data.load_lsmdc_text to:
    keys = ['idx', 'vid', 'vid_start', 'vid_end',
            'cap_start', 'cap_end', 'caption']

code.data.load_fib_text
    keys = ['idx', 'vid', 'full_caption', 'masked_caption',
            'answer', 'vid_key', 'key']         

Even after the following modifications, I still get a key error. How do you suggest rectifying this? In the meantime, I'm trying to remove the index columns, will let you know if I have any updates.

Thank you again for your prompt responses and support.

[dipikakhullar]

Hi @JiwanChung, Just an update: I removed the index columns from the dataset, removed my 'idx' additions to the keys variable, and was able to get past this key error. Now I am getting this error on my machine. Have you seen this before? I tried running with debug =True like you mentioned before, but it doesn't seem to make a difference.

I tried both commands:

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']"

and

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']" --debug=True

Please let me know if this is correct. In addition, I've attached the error traceback. It outputted an error and just hangs at the last line. It was extremely long, so I've attached it as a txt file below.

Thank you, Dipika Khullar Error Report Jan 13.txt

[JiwanChung]

Hi!

The relevant error seems to be the following: - torch_shm_manager: error while loading shared libraries: libnvToolsExt.so.1: cannot open shared object file: No such file or directory This is a CUDA installation error. Though I am not familiar with the exact source of the problem, you may find this issue useful: https://forums.developer.nvidia.com/t/how-can-i-install-the-pytorch/108038

Thank you, Jiwan Chung

On Thu, Jan 13, 2022 at 7:54 PM Dipika Khullar @.***> wrote:

Hi @JiwanChung https://github.com/JiwanChung, Just an update: I removed the index columns from the dataset, removed my 'idx' additions to the keys variable, and was able to get past this key error. Now I am getting this error on my machine. Have you seen this before? I tried running with debug =True like you mentioned before, but it doesn't seem to make a difference.

I tried both commands:

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']"

and

python cli.py train with model=no_gt_sos fix_gpt_epoch=5 feature_names="['video', 'images']" --debug=True

Please let me know if this is correct. In addition, I've attached the error traceback. It outputted an error and just hangs at the last line. It was extremely long, so I've attached it as a txt file below.

Thank you, Dipika Khullar Error Report Jan 13.txt https://github.com/JiwanChung/tapm/files/7862149/Error.Report.Jan.13.txt

— Reply to this email directly, view it on GitHub https://github.com/JiwanChung/tapm/issues/14#issuecomment-1012022229, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE36FWENVVMAD2NBMSPXOETUV2VMRANCNFSM5JVUFJZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[dipikakhullar]

Hi @JiwanChung , I looked at that link, all the dependencies seem to be correct. Do you know what else it might be?

[JiwanChung]

Hi!

I think the problem is related to the difference between CUDA and pytorch versions, according to this link ( https://github.com/MVIG-SJTU/AlphaPose/issues/402). As a possible solution, I suggest commenting out the following line: https://github.com/JiwanChung/tapm/blob/master/code/args.py#L25 Please check if this solves your problem.

On Sat, Jan 29, 2022 at 3:54 PM Dipika Khullar @.***> wrote:

Hi @JiwanChung https://github.com/JiwanChung , I looked at that link, all the dependencies seem to be correct. Do you know what else it might be?

— Reply to this email directly, view it on GitHub https://github.com/JiwanChung/tapm/issues/14#issuecomment-1024851396, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE36FWAE4HQ64TISLIGIATLUYOFMBANCNFSM5JVUFJZA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>