search

FMInference / DejaVu

268 stars 32 forks source link

Compare the sparse model on downstream tasks #10

Open iidsample opened 11 months ago

iidsample commented 11 months ago

Hello Authors,

We have been trying to replicate the results for downstream tasks, based on details here.

However, we are unable to figure out how to specify the sparse-opt-model to run. Being unfamiliar with the code it looks like the current sparse-opt-model does not seem to have integration with the code in Decentralized_FM_alpha. Can you please provide additional information on how can we reproduce the down stream task numbers.

bilgeacun commented 11 months ago

@lzcemma can you help with this please?

lzcemma commented 11 months ago

Hi, Step 1: We use lm-eval-harness to generate test data and save them in json file, as shown in this file

Step 2: Once you have the generated file, simply replace the file path in the script example.

Step 3: The script will automatically save generated test samples in the output path. Please use this file to run evaluate.