Error when wandb is installed

[aclifton314]

System Summary

Pop!_OS 20.04 Pytorch: 1.5.1 Transformers: 3.0.2 Tokenizers: 0.8.1rc1 Python: 3.7.6 Pretrained Model: GPT2 Pretrained Tokenizer: GPT2

Question

Training without wandb works fine but when I pip install wandb and change nothing else in my code, whenever I go to run training I get the following error:

I0821 15:44:17.531560 46912496399424 file_utils.py:39] PyTorch version 1.5.1 available.
I0821 15:44:21.471980 46912496399424 file_utils.py:55] TensorFlow version 2.0.0 available.
Traceback (most recent call last):
  File "run_finetune_gpt2.py", line 5, in <module>
    from transformers import TrainingArguments, Trainer, GPT2Tokenizer
  File "/path/to/venv/my-venv/lib/python3.6/site-packages/transformers/__init__.py", line 158, in <module>
    from .trainer_utils import EvalPrediction, set_seed
  File "/path/to/venv/my-venv/lib/python3.6/site-packages/transformers/trainer_utils.py", line 11, in <module>
    import wandb
  File "/path/to/venv/my-venv/lib/python3.6/site-packages/wandb/__init__.py", line 60, in <module>
    from wandb.apis import InternalApi, PublicApi, CommError
  File "/path/to/venv/my-venv/lib/python3.6/site-packages/wandb/apis/__init__.py", line 116, in <module>
    from .public import Api as PublicApi
  File "/path/to/venv/my-venv/lib/python3.6/site-packages/wandb/apis/public.py", line 28, in <module>
    from wandb.summary import HTTPSummary
  File "/path/to/venv/my-venv/lib/python3.6/site-packages/wandb/summary.py", line 15, in <module>
    from wandb.meta import Meta
  File "/path/to/venv/my-venv/lib/python3.6/site-packages/wandb/meta.py", line 6, in <module>
    import pynvml
  File "/cm/local/apps/cuda/libs/current/pynvml/pynvml.py", line 1671
    print c_count.value
                ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(c_count.value)?

Any thoughts?

[borisdayma]

What version of wandb do you have? Also would you have a simple example to reproduce this issue?

[borisdayma]

It could be a problem with your virtual environment. Maybe this issue will help: https://github.com/wandb/client/issues/539

[aclifton314]

@borisdayma Thank you for your reply. I'm using wand 0.9.5. Here is some sample code:

from transformers import Trainer, TrainingArguments, GPT2LMHeadModel, GPT2Tokenizer
import torch
from torch.utils.data import Dataset

class SDAbstractsDataset(Dataset):
    def __init__(self):
        prompt1 = 'We present an update on the results of the Double Chooz experiment. Double Chooz searches for the neutrino mixing angle, θ13, in the three-neutrino mixing matrix via the disappearance of produced by the dual 4.27 GW/th Chooz B Reactors. Here we discuss updated oscillation fit results using both the rate and the shape of the anti-neutrino energy spectrum. In the most recent oscillation analysis we included data with neutron captures on Gadolinium and Hydrogen along with the reactor off data that we collected. This is an important step in our multi-year program to establish the value of θ13.'
        prompt2 = 'The paper covers detailed discussion on novel control system developed for adaptive fluid-based shock-absorbers serving for mitigation of unknown impact excitations. In order to provide complete independence of the control system from the loading conditions, the Hybrid Prediction Control (HPC) was elaborated. The proposed method is an extension of previously introduced kinematic feedback control which ensures optimal path finding, tracking and path update in case of high disturbance or sudden change of loading conditions. Implementation of the presented control system allows to obtain self-adaptive fluid-based absorbers providing robust impact mitigation. In contrast to previously developed methods of Adaptive Impact Absorption, the proposed control strategy does not require prior knowledge of impact excitation or its preliminary identification. The independence of applied control system from parameters of impact loading results in the capability of automatic path correction in the case of disturbance occurrence and re-adaptation to a number of subsequent impacts. The successful operation of the self-adaptive system is investigated with the use of numerical examples involving double-chamber pneumatic shock-absorber equipped with controllable valve. Efficiency of the HPC is proved by comparison with passive absorber as well as device equipped with adaptive and optimal control modules.'
        prompt3 = 'This study aimed to produce biosurfactant from Pseudozyma tsukubaensis using cassava wastewater and an inoculum (biomass) for galactooligosaccharides synthesis from lactose as an integrated system. First, the use of cassava wastewater as a low cost culture medium by P. tsukubaensis to produce biomass and biosurfactant was evaluated and optimized. Then, the microbial cells (biomass) obtained from the optimized process were used to produce galactooligosaccharides from lactose. The optimum conditions for biosurfactant and biomass synthesis were found to be 80% (v/v) of cassava wastewater at 30°C and 200rpm for 48h. The highest concentration of biosurfactant, that is, minimum surface tension value and maximum biomass concentration predicted were experimentally confirmed as 26.87mN/m and 10.5g/L, respectively. The biosurfactant obtained showed good thermal (121°C/1h), pH (2–11) and ionic strength (0–25% NaCl) stability. Excellent emulsifier activity was also verified, suggesting a potential application in enhanced oil recovery. Galactooligosaccharides synthesized by the Kluyveromyces genus have been extensively investigated, however, few studies have reported transgalactosylation ability by other yeast genera. The transgalactosylation activity of the yeast biomass at optimized conditions from 40% (w/w) lactose resulted in galactooligosaccharides production of 73.12g/L and a yield of 18.28% (w/w) at pH 8.0 and 30°C in 24h. This research showed the technical feasibility of an integrated process: biosurfactant and GOS production from P. tsukubaensis, which takes advantage of the remarkable metabolism of this microorganism. To the best of our knowledge, this is the first study reporting the potential of P. tsukubaensis to produce two economical biotechnological products of increase interest as an integrated process.'
        prompt4 = 'Advantages of a fuzzy predictive control algorithm are discussed in the paper. The fuzzy predictive algorithm is a combination of a DMC (Dynamic Matrix Control) algorithm and Takagi–Sugeno fuzzy modeling, thus it inherits advantages of both techniques. The algorithm is numerically effective. It is in fact generalization of the standard DMC algorithm widely used in the industry, thus the existing implementations of the DMC algorithm can be extended using the presented fuzzy approach. A simple and easy to apply method of fuzzy predictive control algorithms synthesis is presented in the paper. It can be easy applied also in the case of Multiple Input Multiple Output (MIMO) control plants. Moreover, information about measured disturbance can be included in the algorithms in an easy way. The advantages of the fuzzy predictive control algorithm are demonstrated in the example control systems of two nonlinear chemical reactors: the first one—with inverse response and the second one—a MIMO plant with time delay.'
        self.data_list = [prompt1, prompt2, prompt3, prompt4]

    def __len__(self):
        return len(self.data_list)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        abstract_text = self.data_list[idx]
        return abstract_text

def sd_data_collator(dataset_samples_list):
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2', padding_side='right')
    tokenizer.pad_token = tokenizer.eos_token

    encoded_results = tokenizer(dataset_samples_list, padding=True, truncation=True, return_tensors='pt', return_attention_mask=True)

    batch = {}
    batch['input_ids'] = torch.stack([result for result in encoded_results['input_ids']])
    batch['past'] = None
    batch['attention_mask'] = torch.stack([result for result in encoded_results['attention_mask']])
    batch['position_ids'] = None
    batch['head_mask'] = None
    batch['inputs_embeds'] = None
    batch['labels'] = None
    batch['use_cache'] = True
    return batch

output_dir = 'YOUR_OUTPUT_DIR'
logging_dir = 'YOUR_LOGGING_DIR'
training_args = TrainingArguments(
    output_dir=output_dir,
    do_train=True,
    logging_dir=logging_dir,
    save_steps=50,
    per_device_train_batch_size=2
)

model = GPT2LMHeadModel.from_pretrained('gpt2')
sd_dataset = SDAbstractsDataset()

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=sd_dataset,
    data_collator=sd_data_collator
)

#trainer.train()

[borisdayma]

I ran it in colab: see notebook

There does not seem to be any issue related to wandb. Maybe try my previous link as it may be due to issues in your local environment.

[aclifton314]

@borisdayma that link solved my problem! Thanks for your help!