Closed Filco306 closed 2 years ago
How are you generating this config file? And is your model based on the ConvE code somehow? Maybe somehow the ConvE configuration key is still making an appearance somewhere?
On Sat, 25 Sep 2021, 23:57 Filip Cornell, @.***> wrote:
Hello (again, haha)
I am currently running some experiments with your package; I love it. However, I am concerned about the validity of one of my experiments. I performed an AxSearchJob on one of my models (which I have named RIE2). In my config.yaml from one of the runs, I get a conve setting, which is strange to me. I don't know why it is there, and I am concerned that it might be, in the worst case scenario, a ConvE-model training in the background (and not my model implementation).
Now, I have tried to make sure that my worries are not legitimate by introducing exceptions in all parts of the ConvE code, if it is executed, and it seems to run through without a problem. In other words, it does not seem like ConvE is used, since no code from the ConvE-part of the code seems to be used. Also, loading the model will load an RIE2-model.
I would simply like to know; why is there a ConvE part being created in this config.yaml? Even if I erase that part and re-run the experiment, it comes back into the new config.yaml-file for the new experiment. I do not believe there is an issue here, but I would just like to make sure with a 100 % certainty.
Thank you in advance, and thanks again for a very nice package!
The config file is as seen below:
1vsAll: class_name: TrainingJob1vsAllKvsAll: class_name: TrainingJobKvsAll label_smoothing: -0.10110532511025666 query_types: _po: true so: false sp: trueRIE2: class_name: RIE2 entity_embedder: +++: +++ dropout: 0.13526539038866758 normalize: p: 2.0 regularize_weight: 1.4196422325008177e-05 type: lookup_embedder l_norm: 1.0 relation_embedder: +++: +++ dropout: 0.1788316359743476 normalize: p: -1.0 regularize_weight: 0.0002284745422735363 type: lookup_embedder round_dim: false tuneable_doubletan: falseax_search: class_name: AxSearchJob num_sobol_trials: 30 num_trials: 30 parameter_constraints: [] parameters:
- name: model type: choice values:
- RIE2
- reciprocal_relations_model
- is_ordered: true name: train.batch_size type: choice values:
- 128
- 256
- 512
- 1024
- name: train.type type: fixed value: negative_sampling
- name: train.optimizer type: choice values:
- Adam
- Adagrad
- name: train.loss type: fixed value: kl
- bounds:
- 0.0003
- 1.0 log_scale: true name: train.optimizer_args.lr type: range
- name: train.lr_scheduler type: fixed value: ReduceLROnPlateau
- name: train.lr_scheduler_args.mode type: fixed value: max
- name: train.lr_scheduler_args.factor type: fixed value: 0.95
- name: train.lr_scheduler_args.threshold type: fixed value: 0.0001
- bounds:
- 0
- 10 name: train.lr_scheduler_args.patience type: range
- is_ordered: true name: lookup_embedder.dim type: choice values:
- 128
- 256
- 512
- name: lookup_embedder.initialize type: choice values:
- xaviernormal
- xavieruniform
- normal_
- uniform_
- name: lookup_embedder.initializeargs.normal.mean type: fixed value: 0.0
- bounds:
- 1.0e-05
- 1.0 log_scale: true name: lookup_embedder.initializeargs.normal.std type: range
- bounds:
- -1.0
- -1.0e-05 name: lookup_embedder.initializeargs.uniform.a type: range
- name: lookup_embedder.initialize_args.xavieruniform.gain type: fixed value: 1.0
- name: lookup_embedder.initialize_args.xaviernormal.gain type: fixed value: 1.0
- is_ordered: true name: lookup_embedder.regularize type: choice values:
- ''
- l3
- l2
- l1
- name: lookup_embedder.regularize_args.weighted type: choice values:
- true
- false
- bounds:
- 1.0e-20
- 0.1 log_scale: true name: RIE2.entity_embedder.regularize_weight type: range
- bounds:
- 1.0e-20
- 0.1 log_scale: true name: RIE2.relation_embedder.regularize_weight type: range
- bounds:
- -0.5
- 0.5 name: RIE2.entity_embedder.dropout type: range
- bounds:
- -0.5
- 0.5 name: RIE2.relation_embedder.dropout type: range
- bounds:
- -0.3
- 0.3 name: KvsAll.label_smoothing type: range
- is_ordered: true name: RIE2.l_norm type: choice values:
- 1.0
- 2.0
- name: RIE2.entity_embedder.normalize.p type: choice values:
- -1.0
- 2.0
- name: RIE2.relation_embedder.normalize.p type: choice values:
- -1.0
- 2.0
- name: negative_sampling.implementation type: fixed value: spo sobol_seed: 0console: format: {} quiet: falseconve: # <--- HERE IS THE CONVE PART 2D_aspect_ratio: 2 # <--- HERE IS THE CONVE PART class_name: ConvE # <--- HERE IS THE CONVE PART convolution_bias: true # <--- HERE IS THE CONVE PART entity_embedder: # <--- HERE IS THE CONVE PART +++: +++ # <--- HERE IS THE CONVE PART dropout: 0.2 # <--- HERE IS THE CONVE PART type: lookup_embedder # <--- HERE IS THE CONVE PART feature_map_dropout: 0.2 # <--- HERE IS THE CONVE PART filter_size: 3 # <--- HERE IS THE CONVE PART padding: 0 # <--- HERE IS THE CONVE PART projection_dropout: 0.3 # <--- HERE IS THE CONVE PART relation_embedder: # <--- HERE IS THE CONVE PART +++: +++ # <--- HERE IS THE CONVE PART dropout: 0.2 # <--- HERE IS THE CONVE PART type: lookup_embedder # <--- HERE IS THE CONVE PART round_dim: false # <--- HERE IS THE CONVE PART stride: 1 # <--- HERE IS THE CONVE PARTdataset: +++: +++ files: +++: +++ entity_ids: filename: entity_ids.del type: map entity_strings: filename: entity_strings.del type: idmap relation_ids: filename: relation_ids.del type: map relation_strings: filename: relation_ids.del type: map test: filename: test.del size: 3134 split_type: test type: triples test_without_unseen: filename: test_without_unseen.del size: 2924 split_type: test type: triples train: filename: train.del size: 86835 split_type: train type: triples train_sample: filename: train_sample.del size: 3034 split_type: train type: triples valid: filename: valid.del size: 3034 split_type: valid type: triples valid_without_unseen: filename: valid_without_unseen.del size: 2824 split_type: valid type: triples name: wnrr num_entities: 40943 num_relations: 11 pickle: trueentity_ranking: chunk_size: -1 class_name: EntityRankingJob filter_splits:
- train
- valid filter_with_test: true hits_at_k_s:
- 1
- 3
- 10
- 50
- 100
- 200
- 300
- 400
- 500
- 1000 metrics_per: argument_frequency: false head_and_tail: false relation_type: true tie_handling: rounded_mean_rankeval: batch_size: 256 num_workers: 0 pin_memory: false split: valid trace_level: epoch type: entity_rankinggrid_search: class_name: GridSearchJob parameters: +++: +++ run: trueimport:
- RIE2
- reciprocal_relations_modeljob: device: cuda type: trainlookup_embedder: class_name: LookupEmbedder dim: 128 dropout: 0.0 initialize: xavieruniform initializeargs: +++: +++ normal: mean: 0.0 std: 0.00043470941774473033 uniform_: a: -0.4313754705902655 xaviernormal: gain: 1.0 xavieruniform: gain: 1.0 normalize: p: -1.0 pretrain: ensure_all: false model_filename: '' regularize: lp regularize_args: +++: +++ p: 1 weighted: true regularize_weight: 0.0 round_dim_to: [] sparse: falsemanual_search: class_name: ManualSearchJob configurations: [] run: truemodel: RIE2modules:
- kge.model.embedder
- kge.model
- kge.jobnegative_sampling: class_name: TrainingJobNegativeSampling filtering: implementation: fast_if_available o: false p: false s: false split: '' frequency: smoothing: 1 implementation: triple num_samples: o: -1 p: 0 s: 3 sampling_type: uniform shared: false shared_type: default with_replacement: truerandom_seed: default: -1 numba: -1 numpy: -1 python: -1 torch: -1reciprocal_relations_model: base_model: +++: +++ type: RIE2 class_name: ReciprocalRelationsModelsearch: device_pool: [] num_workers: 1 on_error: abort type: ax_searchtrain: abort_on_nan: true auto_correct: true batch_size: 256 checkpoint: every: 5 keep: 3 keep_init: true loss: kl loss_arg: .nan lr_scheduler: ReduceLROnPlateau lr_scheduler_args: +++: +++ factor: 0.95 mode: max patience: 2 threshold: 0.0001 lr_warmup: 0 max_epochs: 400 num_workers: 0 optimizer: +++: +++ default: args: +++: +++ lr: 0.4969193773628293 type: Adagrad pin_memory: false split: train subbatch_auto_tune: false subbatch_size: -1 trace_level: epoch type: negative_sampling visualize_graph: falsetraining_loss: class_name: TrainingLossEvaluationJobuser: +++: +++valid: early_stopping: patience: 10 threshold: epochs: 50 metric_value: 0.05 every: 5 metric: mean_reciprocal_rank_filtered_with_test metric_expr: float("nan") metric_max: true split: valid trace_level: epoch
The new ConvE-file looks like this (so it should not be run, which it does not seem to be).
import torchfrom torch import Tensorimport math from kge import Config, Datasetfrom kge.model.kge_model import RelationalScorer, KgeModel
class ConvEScorer(RelationalScorer): r"""Implementation of the ConvE KGE scorer. Must be used with ReciprocalRelationsModel."""
def __init__(self, config: Config, dataset: Dataset, configuration_key=None): raise Exception("This model should not be used at all! This is the error! ") def score_emb(self, s_emb, p_emb, o_emb, combine: str): raise Exception("This model should not be used at all! This is the error! ")
class ConvE(KgeModel): r"""Implementation of the ConvE KGE model."""
def __init__( self, config: Config, dataset: Dataset, configuration_key=None, init_for_load_only=False, ): raise Exception("This model should not be used at all! This is the error! ") def score_spo(self, s: Tensor, p: Tensor, o: Tensor, direction=None) -> Tensor: raise Exception("This model should not be used at all! This is the error! ")
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/uma-pi1/kge/issues/228, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEWXZGKVOTTFGBE5BTSAKDUDZAUTANCNFSM5EYA6B7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
I generate it using kge start config.yaml
, where config.yaml
is the file above. If I remove the part about ConvE
, it will still be in the new file again which I do not understand why is the case.
And no, I do not base my model on ConvE in any way. I did however copy over some settings for the hyperparameter search from the ICLR config files, but I looked through that and I cannot see the ConvE-name anywhere in that file either.
I think I have figured it out through some backtracing. When I am using the ReciprocalRelationsModel, it will import the ConvE-model
See the reciprocal_relations.yaml
:
import: [conve]
reciprocal_relations_model:
class_name: ReciprocalRelationsModel
base_model:
type: conve
+++: +++
Is this a desired feature? This means that ConvE will always be imported when the reciprocal_relations_model is used. If I understand it correctly, the reciprocal relations model is meant to use different embeddings when predicting subject and object.
In other words, this gets imported automatically, but I believe my model overrides this setting, so the ConvE-model is not used. Please correct me if I am wrong.
This is my hyperparameter search job:
1vsAll:
class_name: TrainingJob1vsAll
KvsAll:
class_name: TrainingJobKvsAll
label_smoothing: 0.0
query_types:
_po: true
s_o: false
sp_: true
RIE2:
class_name: RIE2
entity_embedder:
+++: +++
type: lookup_embedder
l_norm: 1.0
relation_embedder:
+++: +++
type: lookup_embedder
tuneable_doubletan: false
ax_search:
class_name: AxSearchJob
num_sobol_trials: 30
num_trials: 30
parameter_constraints: []
parameters:
- name: model
type: fixed
value: reciprocal_relations_model
- is_ordered: true
name: train.batch_size
type: choice
values:
- 128
- 256
- name: train.type
type: fixed
value: KvsAll
- name: train.optimizer
type: choice
values:
- Adam
- Adagrad
- name: train.loss
type: fixed
value: kl
- bounds:
- 0.0003
- 1.0
log_scale: true
name: train.optimizer_args.lr
type: range
- name: train.lr_scheduler
type: fixed
value: ReduceLROnPlateau
- name: train.lr_scheduler_args.mode
type: fixed
value: max
- name: train.lr_scheduler_args.factor
type: fixed
value: 0.95
- name: train.lr_scheduler_args.threshold
type: fixed
value: 0.0001
- bounds:
- 0
- 10
name: train.lr_scheduler_args.patience
type: range
- is_ordered: true
name: lookup_embedder.dim
type: choice
values:
- 128
- 256
- 512
- name: lookup_embedder.initialize
type: choice
values:
- xavier_normal_
- xavier_uniform_
- normal_
- uniform_
- name: lookup_embedder.initialize_args.normal_.mean
type: fixed
value: 0.0
- bounds:
- 1.0e-05
- 1.0
log_scale: true
name: lookup_embedder.initialize_args.normal_.std
type: range
- bounds:
- -1.0
- -1.0e-05
name: lookup_embedder.initialize_args.uniform_.a
type: range
- name: lookup_embedder.initialize_args.xavier_uniform_.gain
type: fixed
value: 1.0
- name: lookup_embedder.initialize_args.xavier_normal_.gain
type: fixed
value: 1.0
- is_ordered: true
name: lookup_embedder.regularize
type: choice
values:
- ''
- l3
- l2
- l1
- name: lookup_embedder.regularize_args.weighted
type: choice
values:
- true
- false
- bounds:
- 1.0e-20
- 0.1
log_scale: true
name: RIE2.entity_embedder.regularize_weight
type: range
- bounds:
- 1.0e-20
- 0.1
log_scale: true
name: RIE2.relation_embedder.regularize_weight
type: range
- bounds:
- -0.5
- 0.5
name: RIE2.entity_embedder.dropout
type: range
- bounds:
- -0.5
- 0.5
name: RIE2.relation_embedder.dropout
type: range
- bounds:
- -0.3
- 0.3
name: KvsAll.label_smoothing
type: range
sobol_seed: 0
console:
format: {}
quiet: false
conve:
2D_aspect_ratio: 2
class_name: ConvE
convolution_bias: true
entity_embedder:
+++: +++
dropout: 0.2
type: lookup_embedder
feature_map_dropout: 0.2
filter_size: 3
padding: 0
projection_dropout: 0.3
relation_embedder:
+++: +++
dropout: 0.2
type: lookup_embedder
round_dim: false
stride: 1
dataset:
+++: +++
files:
+++: +++
entity_ids:
filename: entity_ids.del
type: map
entity_strings:
filename: entity_ids.del
type: map
relation_ids:
filename: relation_ids.del
type: map
relation_strings:
filename: relation_ids.del
type: map
test:
filename: test.del
type: triples
train:
filename: train.del
type: triples
valid:
filename: valid.del
type: triples
name: wnrr
num_entities: -1
num_relations: -1
pickle: true
entity_ranking:
chunk_size: -1
class_name: EntityRankingJob
filter_splits:
- train
- valid
filter_with_test: true
hits_at_k_s:
- 1
- 3
- 10
- 50
- 100
- 200
- 300
- 400
- 500
- 1000
metrics_per:
argument_frequency: false
head_and_tail: false
relation_type: true
tie_handling: rounded_mean_rank
eval:
batch_size: 16
num_workers: 0
pin_memory: false
split: valid
trace_level: epoch
type: entity_ranking
grid_search:
class_name: GridSearchJob
parameters:
+++: +++
run: true
import:
- RIE2
- reciprocal_relations_model
job:
device: cuda:1
type: search
lookup_embedder:
class_name: LookupEmbedder
dim: 100
dropout: 0.0
initialize: normal_
initialize_args:
+++: +++
normalize:
p: -1.0
pretrain:
ensure_all: false
model_filename: ''
regularize: lp
regularize_args:
+++: +++
p: 2
weighted: false
regularize_weight: 0.0
round_dim_to: []
sparse: false
manual_search:
class_name: ManualSearchJob
configurations: []
run: true
model: ''
modules:
- kge.job
- kge.model.embedder
- kge.model
negative_sampling:
class_name: TrainingJobNegativeSampling
filtering:
implementation: fast_if_available
o: false
p: false
s: false
split: ''
frequency:
smoothing: 1
implementation: batch
num_samples:
o: -1
p: 0
s: 3
sampling_type: uniform
shared: false
shared_type: default
with_replacement: true
random_seed:
default: -1
numba: -1
numpy: -1
python: -1
torch: -1
reciprocal_relations_model:
base_model:
+++: +++
type: RIE2
class_name: ReciprocalRelationsModel
search:
device_pool: []
num_workers: 1
on_error: abort
type: ax_search
train:
abort_on_nan: true
auto_correct: true
batch_size: 100
checkpoint:
every: 5
keep: 3
keep_init: true
loss: kl
loss_arg: .nan
lr_scheduler: ''
lr_scheduler_args:
+++: +++
lr_warmup: 0
max_epochs: 400
num_workers: 0
optimizer:
+++: +++
default:
args:
+++: +++
type: Adagrad
pin_memory: false
split: train
subbatch_auto_tune: false
subbatch_size: -1
trace_level: epoch
type: KvsAll
visualize_graph: false
training_loss:
class_name: TrainingLossEvaluationJob
user:
+++: +++
valid:
early_stopping:
patience: 10
threshold:
epochs: 50
metric_value: 0.05
every: 5
metric: mean_reciprocal_rank_filtered_with_test
metric_expr: float("nan")
metric_max: true
split: valid
trace_level: epoch
As a clarification; If I understand it correctly, this means that the reciprocal relations model is asymmetric, meaning that a prediction of a head is not equal to the prediction of a tail? This has significant meaning in the implementation of my model, so it would be great to know :) Thanks!
The reciprocal relations model uses two different embeddings for each relation. One for predicting subjects and another for predicting objects.
Filip Cornell @.***> schrieb am So. 26. Sept. 2021 um 15:28:
As a clarification; If I understand it correctly, this means that the reciprocal relations model is asymmetric, meaning that a prediction of a head is not equal to the prediction of a tail? This has significant meaning in the implementation of my model, so it would be great to know :) Thanks!
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/uma-pi1/kge/issues/228#issuecomment-927306975, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEWXZE5BHJWKLTZQYPKWC3UD4NXDANCNFSM5EYA6B7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Exactly, just as I understood. Thank you @rufex2001 . Let me know if there is something off with the fact that ConvE is imported; otherwise I will assume it is correct :)
The reciprocal relations model needs a base (scorer) model, which by default is ConvE, so I guess that’s why you imported it
Filip Cornell @.***> schrieb am So. 26. Sept. 2021 um 16:25:
Exactly, just as I understood. Thank yoh @rufex2001 https://github.com/rufex2001 . Let me know if there is something off with the fact that ConvE is imported; otherwise I will assume it is correct :)
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/uma-pi1/kge/issues/228#issuecomment-927314775, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEWXZEY2XRCW44J2PM7V7DUD4UL3ANCNFSM5EYA6B7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Another question while we are at it; the metrics posted in the README for the datasets; which one of the outputs is it? Is it mrr
, mrr_filtered
or mrr_filtered_with_test
that is documented in the README? Thanks again for all your responses!!
Yes, that is my understanding as well; but my model is still used; i.e., the ConvE architecture is not used.
And thanks again for the clarifications!
Results in the README are all mrr_filtered_with_test
Filip Cornell @.***> schrieb am So. 26. Sept. 2021 um 16:31:
And thanks again for the clarifications!
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/uma-pi1/kge/issues/228#issuecomment-927315680, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEWXZCT7KZUNZ5QND5S2QTUD4VCPANCNFSM5EYA6B7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Great, thank you! Will close this now. Thank you for helping me sort this out, but let me know if you believe any issues might be caused by this import anyway. Thank you!
Hello (again, haha)
I am currently running some experiments with your package; I love it. However, I am concerned about the validity of one of my experiments. I performed an AxSearchJob on one of my models (which I have named
RIE2
). In myconfig.yaml
from one of the runs, I get aconve
setting, which is strange to me. I don't know why it is there, and I am concerned that it might be, in the worst case scenario, a ConvE-model training in the background (and not my model implementation).Now, I have tried to make sure that my worries are not legitimate by introducing exceptions in all parts of the ConvE code, if it is executed, and it seems to run through without a problem. In other words, it does not seem like ConvE is used, since no code from the ConvE-part of the code seems to be used. Also, loading the model will load an
RIE2
-model.I would simply like to know; why is there a ConvE part being created in this
config.yaml
? Even if I erase that part and re-run the experiment, it comes back into the newconfig.yaml
-file for the new experiment. I do not believe there is an issue here, but I would just like to make sure with a 100 % certainty.Thank you in advance, and thanks again for a very nice package!
The config file is as seen below:
The new ConvE-file looks like this (so it should not be run, which it does not seem to be).